Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:55:57.808373 30725 test_util.cc:268] Using random seed: -536427801
I20240501 19:55:57.814342 30725 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:57.814393 30725 ts_itest-base.cc:113] --------------
I20240501 19:55:57.814411 30725 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:55:57.814427 30725 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:57.814440 30725 ts_itest-base.cc:116] --------------
2024-05-01T19:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:57Z Disabled control of system clock
I20240501 19:55:57.822919 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:46341
--webserver_interface=127.30.1.126
--webserver_port=0
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.1.126:46341
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:55:57.920150 30737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:57.920351 30737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:57.920419 30737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:57.923331 30737 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:57.923401 30737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:57.923449 30737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:57.923485 30737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:57.923533 30737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:57.926965 30737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.1.126:46341
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.1.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:57.927724 30737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.928733 30737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:55:57.936026 30737 server_base.cc:1034] running on GCE node
W20240501 19:55:57.936697 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.937140 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.937800 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.938206 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:57.939005 30737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:57.940150 30737 hybrid_clock.cc:648] HybridClock initialized: now 1714593357940136 us; error 31 us; skew 500 ppm
I20240501 19:55:57.941773 30737 webserver.cc:460] Webserver started at http://127.30.1.126:40797/ using document root <none> and password file <none>
I20240501 19:55:57.942268 30737 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.942319 30737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.942473 30737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.947144 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b338b2d6c2ab408985d76314b5f48e7c"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-htmr"
I20240501 19:55:57.947587 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b338b2d6c2ab408985d76314b5f48e7c"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-htmr"
I20240501 19:55:57.950686 30737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:57.952601 30751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.953537 30737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.953649 30737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b338b2d6c2ab408985d76314b5f48e7c"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-htmr"
I20240501 19:55:57.953722 30737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.978716 30737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.979209 30737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.979359 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.985435 30737 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:46341
I20240501 19:55:57.985455 30803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:46341 every 8 connection(s)
I20240501 19:55:57.986258 30737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:57.988035 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 30737
I20240501 19:55:57.988163 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:57.988981 30804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:57.994829 30804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c: Bootstrap starting.
I20240501 19:55:57.997037 30804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.997706 30804 log.cc:824] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:57.999444 30804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c: No bootstrap required, opened a new log
I20240501 19:55:58.002311 30804 raft_consensus.cc:348] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } }
I20240501 19:55:58.002504 30804 raft_consensus.cc:374] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.002569 30804 raft_consensus.cc:729] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b338b2d6c2ab408985d76314b5f48e7c, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.002974 30804 consensus_queue.cc:260] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } }
I20240501 19:55:58.003124 30804 raft_consensus.cc:388] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:58.003187 30804 raft_consensus.cc:482] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:58.003283 30804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.003950 30804 raft_consensus.cc:504] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } }
I20240501 19:55:58.004221 30804 leader_election.cc:302] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b338b2d6c2ab408985d76314b5f48e7c; no voters:
I20240501 19:55:58.004427 30804 leader_election.cc:288] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:58.004772 30809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:58.004966 30809 raft_consensus.cc:686] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [term 1 LEADER]: Becoming Leader. State: Replica: b338b2d6c2ab408985d76314b5f48e7c, State: Running, Role: LEADER
I20240501 19:55:58.005205 30809 consensus_queue.cc:237] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } }
I20240501 19:55:58.005375 30804 sys_catalog.cc:564] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:58.006508 30810 sys_catalog.cc:455] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b338b2d6c2ab408985d76314b5f48e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } } }
I20240501 19:55:58.006604 30810 sys_catalog.cc:458] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:58.006546 30811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b338b2d6c2ab408985d76314b5f48e7c. Latest consensus state: current_term: 1 leader_uuid: "b338b2d6c2ab408985d76314b5f48e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b338b2d6c2ab408985d76314b5f48e7c" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 46341 } } }
I20240501 19:55:58.006947 30818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:58.006968 30811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:58.009406 30818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:58.013624 30818 catalog_manager.cc:1347] Generated new cluster ID: 8e58f41fd8824cae8243c722bfa44852
I20240501 19:55:58.013731 30818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:58.028599 30818 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:58.029392 30818 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:58.037281 30818 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b338b2d6c2ab408985d76314b5f48e7c: Generated new TSK 0
I20240501 19:55:58.037809 30818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:58.048444 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.65:0
--local_ip_for_outbound_sockets=127.30.1.65
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 19:55:58.148751 30828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.148928 30828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.148988 30828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.151649 30828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.151703 30828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.151783 30828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.65
I20240501 19:55:58.155122 30828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.1.65
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.30828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.65
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.155812 30828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.156867 30828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.164391 30836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.164876 30833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.165580 30834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.166378 30828 server_base.cc:1034] running on GCE node
I20240501 19:55:58.166666 30828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.167090 30828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.168243 30828 hybrid_clock.cc:648] HybridClock initialized: now 1714593358168215 us; error 35 us; skew 500 ppm
I20240501 19:55:58.169957 30828 webserver.cc:460] Webserver started at http://127.30.1.65:33291/ using document root <none> and password file <none>
I20240501 19:55:58.170423 30828 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.170471 30828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.170610 30828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.172008 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.172439 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.175472 30828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 19:55:58.177323 30842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.178201 30828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.178303 30828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.178367 30828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.202229 30828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.202708 30828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.202823 30828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.203245 30828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.204013 30828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.204057 30828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.204092 30828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.204110 30828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.212361 30828 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.65:43707
I20240501 19:55:58.212391 30955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.65:43707 every 8 connection(s)
I20240501 19:55:58.213214 30828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:58.213691 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 30828
I20240501 19:55:58.213817 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:58.215821 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.66:0
--local_ip_for_outbound_sockets=127.30.1.66
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:58.223218 30956 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:58.223408 30956 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.223848 30956 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:58.225257 30768 ts_manager.cc:194] Registered new tserver with Master: 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:55:58.226356 30768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.65:55911
W20240501 19:55:58.316265 30959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.316469 30959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.316541 30959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.319341 30959 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.319401 30959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.319504 30959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.66
I20240501 19:55:58.323062 30959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.1.66
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.30959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.66
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.323803 30959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.324705 30959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.332160 30965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.332603 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.333374 30964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.334002 30959 server_base.cc:1034] running on GCE node
I20240501 19:55:58.334318 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.334798 30959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.335950 30959 hybrid_clock.cc:648] HybridClock initialized: now 1714593358335933 us; error 27 us; skew 500 ppm
I20240501 19:55:58.338145 30959 webserver.cc:460] Webserver started at http://127.30.1.66:34555/ using document root <none> and password file <none>
I20240501 19:55:58.338850 30959 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.338929 30959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.339229 30959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.341241 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "17d51481decd4d06964f09c66f981e09"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.341804 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "17d51481decd4d06964f09c66f981e09"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.345081 30959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:58.346968 30973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.347822 30959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.347925 30959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "17d51481decd4d06964f09c66f981e09"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.348001 30959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.367372 30959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.367877 30959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.368013 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.368453 30959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.369237 30959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.369284 30959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.369335 30959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.369364 30959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.377292 30959 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.66:37547
I20240501 19:55:58.377324 31086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.66:37547 every 8 connection(s)
I20240501 19:55:58.378067 30959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:58.381037 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 30959
I20240501 19:55:58.381165 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:58.383285 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.67:0
--local_ip_for_outbound_sockets=127.30.1.67
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:58.386332 31087 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:58.386531 31087 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.386955 31087 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:58.387820 30768 ts_manager.cc:194] Registered new tserver with Master: 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:55:58.388345 30768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.66:33425
W20240501 19:55:58.480100 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.480278 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.480352 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.483292 31090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.483356 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.483448 31090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.67
I20240501 19:55:58.486707 31090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.1.67
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.67
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.487380 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.488302 31090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.495422 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.496102 31096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.496872 31098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.497524 31090 server_base.cc:1034] running on GCE node
I20240501 19:55:58.497872 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.498333 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.499481 31090 hybrid_clock.cc:648] HybridClock initialized: now 1714593358499467 us; error 31 us; skew 500 ppm
I20240501 19:55:58.501083 31090 webserver.cc:460] Webserver started at http://127.30.1.67:37147/ using document root <none> and password file <none>
I20240501 19:55:58.501551 31090 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.501608 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.501797 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.503202 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ac3d66d1d57441db81def86faef0eb4a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.503654 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ac3d66d1d57441db81def86faef0eb4a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.506664 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:58.508555 31104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.509363 31090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.509462 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ac3d66d1d57441db81def86faef0eb4a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.509546 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.520474 31090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.520990 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.521142 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.521575 31090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.522325 31090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.522373 31090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.522425 31090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.522459 31090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.530329 31090 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.67:36463
I20240501 19:55:58.530354 31217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.67:36463 every 8 connection(s)
I20240501 19:55:58.531095 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:58.538260 31218 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:58.538441 31218 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.538441 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31090
I20240501 19:55:58.538512 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:58.538825 31218 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:58.539695 30767 ts_manager.cc:194] Registered new tserver with Master: ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:55:58.540254 30767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.67:56051
I20240501 19:55:58.540263 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.68:0
--local_ip_for_outbound_sockets=127.30.1.68
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 19:55:58.637532 31221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.637710 31221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.637753 31221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.640679 31221 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.640733 31221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.640810 31221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.68
I20240501 19:55:58.644210 31221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.1.68
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.68
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.644937 31221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.645864 31221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.653509 31229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.654138 31227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.654578 31226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.655308 31221 server_base.cc:1034] running on GCE node
I20240501 19:55:58.655637 31221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.656105 31221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.657256 31221 hybrid_clock.cc:648] HybridClock initialized: now 1714593358657244 us; error 24 us; skew 500 ppm
I20240501 19:55:58.658838 31221 webserver.cc:460] Webserver started at http://127.30.1.68:37741/ using document root <none> and password file <none>
I20240501 19:55:58.659299 31221 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.659354 31221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.659504 31221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.660859 31221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5530341eedde430fbacb4c3ff815159c"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.661332 31221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5530341eedde430fbacb4c3ff815159c"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.664172 31221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:55:58.666095 31235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.666935 31221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:58.667076 31221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5530341eedde430fbacb4c3ff815159c"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.667214 31221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.678279 31221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.678740 31221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.678870 31221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.679319 31221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.680045 31221 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.680088 31221 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.680123 31221 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.680159 31221 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.687768 31221 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.68:37085
I20240501 19:55:58.687808 31348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.68:37085 every 8 connection(s)
I20240501 19:55:58.688514 31221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:58.695255 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31221
I20240501 19:55:58.695366 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:58.696019 31349 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:58.696210 31349 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.696622 31349 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:58.697365 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:0
--local_ip_for_outbound_sockets=127.30.1.69
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:58.697551 30767 ts_manager.cc:194] Registered new tserver with Master: 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:55:58.698272 30767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.68:44273
W20240501 19:55:58.796746 31352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.796937 31352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.796983 31352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.799805 31352 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.799857 31352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.799940 31352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:55:58.803318 31352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.803936 31352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.804795 31352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.811821 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.812541 31358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.813200 31360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.813879 31352 server_base.cc:1034] running on GCE node
I20240501 19:55:58.814205 31352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.814702 31352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.815843 31352 hybrid_clock.cc:648] HybridClock initialized: now 1714593358815822 us; error 33 us; skew 500 ppm
I20240501 19:55:58.817471 31352 webserver.cc:460] Webserver started at http://127.30.1.69:36021/ using document root <none> and password file <none>
I20240501 19:55:58.817910 31352 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.817960 31352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.818114 31352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.819514 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3393eeb225a4430382811e9bef2948fe"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.819916 31352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3393eeb225a4430382811e9bef2948fe"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.822753 31352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240501 19:55:58.824493 31366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.825367 31352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.825465 31352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3393eeb225a4430382811e9bef2948fe"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.825534 31352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.855827 31352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.856335 31352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.856465 31352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.856890 31352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.857676 31352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.857719 31352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.857753 31352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.857769 31352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.865497 31352 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:45745
I20240501 19:55:58.865537 31479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:45745 every 8 connection(s)
I20240501 19:55:58.866250 31352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:58.872473 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31352
I20240501 19:55:58.872588 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:58.873886 31480 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:58.874094 31480 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.874533 31480 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:58.874542 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:0
--local_ip_for_outbound_sockets=127.30.1.70
--webserver_interface=127.30.1.70
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:58.875367 30767 ts_manager.cc:194] Registered new tserver with Master: 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745)
I20240501 19:55:58.875845 30767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:39761
W20240501 19:55:58.971714 31483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.971906 31483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.971977 31483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.974844 31483 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.974905 31483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.974994 31483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:55:58.978391 31483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.979091 31483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.980003 31483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.987857 31489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.988590 31491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.989092 31488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.989795 31483 server_base.cc:1034] running on GCE node
I20240501 19:55:58.990093 31483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.990541 31483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.991691 31483 hybrid_clock.cc:648] HybridClock initialized: now 1714593358991681 us; error 28 us; skew 500 ppm
I20240501 19:55:58.993566 31483 webserver.cc:460] Webserver started at http://127.30.1.70:37691/ using document root <none> and password file <none>
I20240501 19:55:58.994197 31483 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.994285 31483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.994486 31483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.996071 31483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:58.996568 31483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:59.000662 31483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 19:55:59.003352 31497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.004789 31483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:59.004961 31483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:55:59.005146 31483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.027887 31483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.028419 31483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.028554 31483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.029059 31483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.029845 31483 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.029891 31483 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.029925 31483 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.029946 31483 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.037766 31483 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:39065
I20240501 19:55:59.037851 31610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:39065 every 8 connection(s)
I20240501 19:55:59.038573 31483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:55:59.039249 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31483
I20240501 19:55:59.039352 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:55:59.041412 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:0
--local_ip_for_outbound_sockets=127.30.1.71
--webserver_interface=127.30.1.71
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:59.048122 31611 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:59.048293 31611 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.048709 31611 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:59.049669 30767 ts_manager.cc:194] Registered new tserver with Master: 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065)
I20240501 19:55:59.050415 30767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:59725
W20240501 19:55:59.141328 31614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.141515 31614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.141556 31614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.144315 31614 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.144363 31614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.144421 31614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:55:59.147702 31614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.148294 31614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.149150 31614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.156625 31619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.157214 31622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.157649 31620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.158299 31614 server_base.cc:1034] running on GCE node
I20240501 19:55:59.158605 31614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.159014 31614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.160172 31614 hybrid_clock.cc:648] HybridClock initialized: now 1714593359160157 us; error 26 us; skew 500 ppm
I20240501 19:55:59.161782 31614 webserver.cc:460] Webserver started at http://127.30.1.71:32975/ using document root <none> and password file <none>
I20240501 19:55:59.162243 31614 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.162297 31614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.162464 31614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.163853 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "af53e6b6c5e549dda69f656f9a755c5b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.164239 31614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "af53e6b6c5e549dda69f656f9a755c5b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.167229 31614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:59.169041 31628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.169883 31614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.169978 31614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "af53e6b6c5e549dda69f656f9a755c5b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.170053 31614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.196421 31614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.196967 31614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.197108 31614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.197595 31614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.198367 31614 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.198423 31614 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.198477 31614 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.198510 31614 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.206781 31614 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:37429
I20240501 19:55:59.206874 31741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:37429 every 8 connection(s)
I20240501 19:55:59.207602 31614 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:55:59.215533 31742 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:59.215701 31742 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.216091 31742 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:59.216989 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31614
I20240501 19:55:59.217108 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:55:59.217090 30767 ts_manager.cc:194] Registered new tserver with Master: af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429)
I20240501 19:55:59.217708 30767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:38939
I20240501 19:55:59.219142 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.72:0
--local_ip_for_outbound_sockets=127.30.1.72
--webserver_interface=127.30.1.72
--webserver_port=0
--tserver_master_addrs=127.30.1.126:46341
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:55:59.228258 30956 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
W20240501 19:55:59.321229 31745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.321393 31745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.321434 31745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.324226 31745 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.324312 31745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.324391 31745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.72
I20240501 19:55:59.327697 31745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.1.72
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.31745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.72
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.328346 31745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.329319 31745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.336640 31751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.337327 31750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.338037 31753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.338702 31745 server_base.cc:1034] running on GCE node
I20240501 19:55:59.339020 31745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.339460 31745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.340602 31745 hybrid_clock.cc:648] HybridClock initialized: now 1714593359340583 us; error 33 us; skew 500 ppm
I20240501 19:55:59.342187 31745 webserver.cc:460] Webserver started at http://127.30.1.72:40187/ using document root <none> and password file <none>
I20240501 19:55:59.342653 31745 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.342715 31745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.342869 31745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.344302 31745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a159fd4137c34800b2f1d142e2272705"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.344713 31745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a159fd4137c34800b2f1d142e2272705"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.347555 31745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:55:59.349387 31759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.350211 31745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.350309 31745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a159fd4137c34800b2f1d142e2272705"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:55:59.350382 31745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.372723 31745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.373217 31745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.373370 31745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.373761 31745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.374461 31745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.374504 31745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.374537 31745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.374557 31745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.385216 31745 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.72:45409
I20240501 19:55:59.385262 31872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.72:45409 every 8 connection(s)
I20240501 19:55:59.385934 31745 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:55:59.390020 31087 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:55:59.393419 31873 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:55:59.393568 31873 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.393891 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 31745
I20240501 19:55:59.393951 31873 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:55:59.393988 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:55:59.394726 30768 ts_manager.cc:194] Registered new tserver with Master: a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:55:59.395280 30768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.72:54063
I20240501 19:55:59.396272 30725 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:55:59.406265 30725 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:55:59.406330 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31352
I20240501 19:55:59.411628 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31483
I20240501 19:55:59.418715 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31614
I20240501 19:55:59.423770 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31745
I20240501 19:55:59.542161 31218 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:55:59.699867 31349 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:03.181138 30767 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:03.201189 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 940fb959cd9248508c1fa45c7ed84385 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.201189 30890 tablet_service.cc:1467] Processing CreateTablet for tablet 9de501bd2ff545f9842d386a2ea8fba8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.201189 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7d50279364b53953d0a0d45edfb9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.201189 31283 tablet_service.cc:1467] Processing CreateTablet for tablet 940fb959cd9248508c1fa45c7ed84385 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.202131 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de501bd2ff545f9842d386a2ea8fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.202131 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940fb959cd9248508c1fa45c7ed84385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.202138 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.202433 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940fb959cd9248508c1fa45c7ed84385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.204281 30889 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7d50279364b53953d0a0d45edfb9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.204530 30889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.205193 31020 tablet_service.cc:1467] Processing CreateTablet for tablet 9de501bd2ff545f9842d386a2ea8fba8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.205413 31020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de501bd2ff545f9842d386a2ea8fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.209295 31895 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:03.209437 31019 tablet_service.cc:1467] Processing CreateTablet for tablet 33c7d50279364b53953d0a0d45edfb9a (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.209661 31019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.209404 31151 tablet_service.cc:1467] Processing CreateTablet for tablet 9de501bd2ff545f9842d386a2ea8fba8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.210158 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de501bd2ff545f9842d386a2ea8fba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.210227 31894 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:03.211294 31895 tablet_bootstrap.cc:654] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.211643 31152 tablet_service.cc:1467] Processing CreateTablet for tablet 940fb959cd9248508c1fa45c7ed84385 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.211863 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940fb959cd9248508c1fa45c7ed84385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.212383 31893 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:03.212577 31895 log.cc:824] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.214272 31895 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:03.214274 31893 tablet_bootstrap.cc:654] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.212030 31894 tablet_bootstrap.cc:654] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.214412 31895 ts_tablet_manager.cc:1367] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 19:56:03.215015 31894 log.cc:824] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.216576 31893 log.cc:824] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.217242 31897 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:03.217819 31895 raft_consensus.cc:348] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.218041 31895 raft_consensus.cc:374] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.218102 31895 raft_consensus.cc:729] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.218531 31895 consensus_queue.cc:260] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.218962 31897 tablet_bootstrap.cc:654] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.219341 31895 ts_tablet_manager.cc:1398] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240501 19:56:03.219570 31895 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:03.219626 31897 log.cc:824] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.220464 31895 tablet_bootstrap.cc:654] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.221201 31894 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:03.221482 31894 ts_tablet_manager.cc:1367] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 19:56:03.222787 31897 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:03.222932 31897 ts_tablet_manager.cc:1367] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 19:56:03.223064 31893 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:03.223273 31893 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 19:56:03.225175 31894 raft_consensus.cc:348] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.225356 31894 raft_consensus.cc:374] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.225416 31894 raft_consensus.cc:729] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.225822 31894 consensus_queue.cc:260] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.225903 31893 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.226078 31893 raft_consensus.cc:374] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.226156 31893 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.226511 31893 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.226560 31894 ts_tablet_manager.cc:1398] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240501 19:56:03.226454 31897 raft_consensus.cc:348] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.226613 31897 raft_consensus.cc:374] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.226672 31897 raft_consensus.cc:729] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.226861 31894 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:03.227118 31893 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:03.227412 31893 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:03.227660 31894 tablet_bootstrap.cc:654] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.227994 31895 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:03.228058 31895 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 19:56:03.228369 31893 tablet_bootstrap.cc:654] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.228574 31895 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.228631 31894 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:03.228663 31895 raft_consensus.cc:374] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.228695 31894 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:03.228693 31895 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.228803 31895 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.229044 31895 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:03.229185 31895 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:03.229169 31894 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.229264 31894 raft_consensus.cc:374] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.229292 31894 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.229390 31894 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.229696 31894 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.229969 31895 tablet_bootstrap.cc:654] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.231024 31897 consensus_queue.cc:260] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.231849 31897 ts_tablet_manager.cc:1398] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
I20240501 19:56:03.232185 31897 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:03.232672 31895 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:03.232753 31895 ts_tablet_manager.cc:1367] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:03.233048 31897 tablet_bootstrap.cc:654] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.233067 31893 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:03.233196 31895 raft_consensus.cc:348] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.233248 31893 ts_tablet_manager.cc:1367] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:03.233294 31895 raft_consensus.cc:374] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.233333 31895 raft_consensus.cc:729] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.233453 31895 consensus_queue.cc:260] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.233736 31895 ts_tablet_manager.cc:1398] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.234285 31897 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:03.234246 31893 raft_consensus.cc:348] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.234364 31897 ts_tablet_manager.cc:1367] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:03.234391 31893 raft_consensus.cc:374] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.234460 31893 raft_consensus.cc:729] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.234578 31893 consensus_queue.cc:260] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.234885 31893 ts_tablet_manager.cc:1398] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.234851 31897 raft_consensus.cc:348] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.234952 31897 raft_consensus.cc:374] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.234988 31897 raft_consensus.cc:729] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.235133 31897 consensus_queue.cc:260] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.235388 31897 ts_tablet_manager.cc:1398] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 19:56:03.284862 31219 tablet.cc:2369] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.324569 31900 raft_consensus.cc:482] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.324752 31900 raft_consensus.cc:504] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.325927 31904 raft_consensus.cc:482] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.326076 31900 leader_election.cc:288] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.326066 31904 raft_consensus.cc:504] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.327100 31904 leader_election.cc:288] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:03.330919 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:03.331101 30910 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:03.331558 30975 leader_election.cc:302] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:03.331739 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:03.331916 31900 raft_consensus.cc:2781] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.331972 31303 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:03.331993 31900 raft_consensus.cc:482] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.332036 31900 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.332978 31900 raft_consensus.cc:504] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.333245 31900 leader_election.cc:288] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.333468 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09" is_pre_election: true
I20240501 19:56:03.333524 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:03.333622 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:03.333619 30910 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.333678 31041 raft_consensus.cc:2379] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in current term 1: Already voted for candidate 17d51481decd4d06964f09c66f981e09 in this term.
I20240501 19:56:03.333742 31303 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.334046 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:03.334619 30910 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.334884 31303 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.335002 30846 leader_election.cc:302] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543; no voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c
I20240501 19:56:03.335197 30977 leader_election.cc:302] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:03.335225 31904 raft_consensus.cc:2726] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:03.335348 31900 raft_consensus.cc:2781] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.335523 31900 raft_consensus.cc:686] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:03.335803 31900 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:03.338127 30766 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:03.359594 31900 raft_consensus.cc:482] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.359676 31900 raft_consensus.cc:504] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.360026 31900 leader_election.cc:288] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:03.360544 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de501bd2ff545f9842d386a2ea8fba8" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:03.360675 30910 raft_consensus.cc:2454] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:03.360947 30975 leader_election.cc:302] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:03.361099 31900 raft_consensus.cc:2781] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.361160 31900 raft_consensus.cc:482] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.361207 31900 raft_consensus.cc:3037] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.361970 31900 raft_consensus.cc:504] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.362241 31900 leader_election.cc:288] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:03.362557 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de501bd2ff545f9842d386a2ea8fba8" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:03.362660 30910 raft_consensus.cc:3037] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.363296 30910 raft_consensus.cc:2454] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.363428 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de501bd2ff545f9842d386a2ea8fba8" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:03.363425 31172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de501bd2ff545f9842d386a2ea8fba8" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:03.363554 30975 leader_election.cc:302] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:03.363652 31171 raft_consensus.cc:3037] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.363695 31900 raft_consensus.cc:2781] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.363756 31900 raft_consensus.cc:686] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:03.363874 31900 consensus_queue.cc:237] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.364706 31171 raft_consensus.cc:2454] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.365325 30766 catalog_manager.cc:5484] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:03.382041 31088 tablet.cc:2369] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.389374 31916 raft_consensus.cc:482] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.389448 31916 raft_consensus.cc:504] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.389767 31916 leader_election.cc:288] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:03.390075 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:03.390071 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:03.390173 31303 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:03.390180 31171 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:03.390421 30975 leader_election.cc:302] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:03.390596 31916 raft_consensus.cc:2781] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.390657 31916 raft_consensus.cc:482] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.390686 31916 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.391351 31916 raft_consensus.cc:504] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.391650 31916 leader_election.cc:288] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:03.391928 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:03.391978 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:03.392020 31303 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.392099 31171 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.392706 31303 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.392741 31171 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:03.392976 30977 leader_election.cc:302] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:03.393114 31916 raft_consensus.cc:2781] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.393265 31916 raft_consensus.cc:686] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:03.393385 31916 consensus_queue.cc:237] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.394961 30766 catalog_manager.cc:5484] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:03.443156 31350 tablet.cc:2369] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:03.467331 30957 tablet.cc:2369] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:03.540017 31920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.540256 31920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.543406 31920 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:03.627061 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.627784 30910 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.627902 31171 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.628527 31916 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.628640 31900 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.628921 31916 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.629282 31303 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.629597 31916 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.637784 30909 raft_consensus.cc:1261] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.637868 31172 raft_consensus.cc:1261] T 9de501bd2ff545f9842d386a2ea8fba8 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.638106 31916 consensus_queue.cc:1035] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.638368 31916 consensus_queue.cc:1035] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.644516 31947 mvcc.cc:205] Tried to move back new op lower bound from 7022974417410101248 to 7022974416225517568. Current Snapshot: MvccSnapshot[applied={T|T < 7022974417410101248}]
I20240501 19:56:03.645556 31951 mvcc.cc:205] Tried to move back new op lower bound from 7022974417410101248 to 7022974416225517568. Current Snapshot: MvccSnapshot[applied={T|T < 7022974417410101248}]
I20240501 19:56:03.657142 31950 mvcc.cc:205] Tried to move back new op lower bound from 7022974417410101248 to 7022974416225517568. Current Snapshot: MvccSnapshot[applied={T|T < 7022974417410101248}]
I20240501 19:56:03.672981 31958 mvcc.cc:205] Tried to move back new op lower bound from 7022974417413906432 to 7022974416460525568. Current Snapshot: MvccSnapshot[applied={T|T < 7022974417413906432}]
INSERT report
rows total: 2000
time total: 110.46 ms
time per row: 0.0552301 ms
I20240501 19:56:03.691926 30761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:03.696848 31151 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccd7f3fea4473497944de737911828 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.696961 31283 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccd7f3fea4473497944de737911828 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.697147 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ccd7f3fea4473497944de737911828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.697202 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ccd7f3fea4473497944de737911828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.698987 31894 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:03.699465 31897 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:03.699527 31283 tablet_service.cc:1467] Processing CreateTablet for tablet bb93992159a3461b945d31da3c9bb43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.699728 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.699795 31894 tablet_bootstrap.cc:654] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.700158 31897 tablet_bootstrap.cc:654] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.701073 31897 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:03.701094 31021 tablet_service.cc:1467] Processing CreateTablet for tablet dbe3171686254a27a3bbe08786fa7503 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.701153 31897 ts_tablet_manager.cc:1367] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:03.701313 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe3171686254a27a3bbe08786fa7503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.701582 31897 raft_consensus.cc:348] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.701655 31897 raft_consensus.cc:374] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.701674 31897 raft_consensus.cc:729] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.701769 31897 consensus_queue.cc:260] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.702047 31897 ts_tablet_manager.cc:1398] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:03.703140 31895 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:03.703897 31895 tablet_bootstrap.cc:654] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.704326 30890 tablet_service.cc:1467] Processing CreateTablet for tablet b2ccd7f3fea4473497944de737911828 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.704547 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2ccd7f3fea4473497944de737911828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.705185 30889 tablet_service.cc:1467] Processing CreateTablet for tablet bb93992159a3461b945d31da3c9bb43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.705360 30889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.706871 31893 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:03.707059 30888 tablet_service.cc:1467] Processing CreateTablet for tablet dbe3171686254a27a3bbe08786fa7503 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.707305 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe3171686254a27a3bbe08786fa7503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.707633 31893 tablet_bootstrap.cc:654] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.708537 31893 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:03.708613 31893 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.709045 31893 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.709144 31893 raft_consensus.cc:374] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.709173 31893 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.709165 31151 tablet_service.cc:1467] Processing CreateTablet for tablet bb93992159a3461b945d31da3c9bb43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.709278 31893 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.709393 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.709508 31893 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.709669 31893 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:03.710455 31893 tablet_bootstrap.cc:654] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.710855 31897 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:03.710960 31894 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:03.711045 31894 ts_tablet_manager.cc:1367] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 19:56:03.711024 31152 tablet_service.cc:1467] Processing CreateTablet for tablet dbe3171686254a27a3bbe08786fa7503 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.711236 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe3171686254a27a3bbe08786fa7503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.711520 31894 raft_consensus.cc:348] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.711606 31894 raft_consensus.cc:374] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.711637 31894 raft_consensus.cc:729] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.711647 31897 tablet_bootstrap.cc:654] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.711738 31894 consensus_queue.cc:260] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.711989 31894 ts_tablet_manager.cc:1398] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.712167 31894 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:03.713182 31894 tablet_bootstrap.cc:654] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.713529 31897 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:03.713585 31893 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:03.713595 31897 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:03.713675 31893 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:03.713996 31897 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.714075 31897 raft_consensus.cc:374] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.714073 31893 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.714118 31897 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.714149 31893 raft_consensus.cc:374] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.714174 31895 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:03.714176 31893 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.714234 31895 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:03.714215 31897 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.714270 31893 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.714439 31897 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:03.714468 31893 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.714576 31897 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:03.714586 31893 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:03.714607 31895 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.714677 31895 raft_consensus.cc:374] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.714722 31895 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.714813 31895 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.715044 31895 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.715296 31893 tablet_bootstrap.cc:654] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.715378 31897 tablet_bootstrap.cc:654] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.716286 31894 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:03.716362 31894 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:03.716785 31894 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.717260 31897 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:03.717281 31893 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:03.717352 31893 ts_tablet_manager.cc:1367] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:03.717341 31897 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:03.716889 31894 raft_consensus.cc:374] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.717614 31894 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.717721 31894 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.717761 31893 raft_consensus.cc:348] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.717831 31893 raft_consensus.cc:374] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.717860 31893 raft_consensus.cc:729] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.717811 31897 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.717906 31897 raft_consensus.cc:374] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.717950 31897 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.717994 31894 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:03.717978 31893 consensus_queue.cc:260] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.718045 31897 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.718166 31893 ts_tablet_manager.cc:1398] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.718254 31897 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.787842 31904 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.787931 31904 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.788324 31904 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.788642 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:03.788766 31302 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:03.789044 30846 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:03.789211 31904 raft_consensus.cc:2781] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.789273 31904 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.789322 31904 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.790243 31904 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.790526 31904 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.790966 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:03.791064 31302 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.791890 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:03.791893 31172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:03.792042 31171 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:03.792294 31302 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 1.
I20240501 19:56:03.792544 30846 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c; no voters: ac3d66d1d57441db81def86faef0eb4a
I20240501 19:56:03.792704 31904 raft_consensus.cc:2781] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.792868 31904 raft_consensus.cc:686] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 LEADER]: Becoming Leader. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Running, Role: LEADER
I20240501 19:56:03.793164 31904 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.794981 30764 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:03.821957 31909 raft_consensus.cc:482] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.822055 31909 raft_consensus.cc:504] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.822927 31909 leader_election.cc:288] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:03.826313 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:03.826321 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09" is_pre_election: true
I20240501 19:56:03.826416 30910 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:03.826450 31041 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:03.826719 31106 leader_election.cc:302] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:03.826918 31909 raft_consensus.cc:2781] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.826979 31909 raft_consensus.cc:482] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.827006 31909 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.827728 31909 raft_consensus.cc:504] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.828023 31909 leader_election.cc:288] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Requested vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:03.828315 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:03.828332 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:03.828414 31041 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.828410 30910 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.829082 30910 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:03.829135 31041 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:03.829363 31106 leader_election.cc:302] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:03.829535 31909 raft_consensus.cc:2781] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.829689 31909 raft_consensus.cc:686] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:03.829918 31909 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:03.831650 30764 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:03.907202 31904 raft_consensus.cc:482] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.907291 31904 raft_consensus.cc:504] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.907696 31904 leader_election.cc:288] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.908023 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:03.908105 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:03.908151 31171 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:03.908231 31302 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:03.908419 30844 leader_election.cc:302] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:03.908636 31904 raft_consensus.cc:2781] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.908681 31904 raft_consensus.cc:482] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.908727 31904 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.909487 31904 raft_consensus.cc:504] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.909785 31904 leader_election.cc:288] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:03.910110 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:03.910115 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:03.910203 31302 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.910243 31171 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.910928 31171 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 1.
I20240501 19:56:03.910929 31302 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 1.
I20240501 19:56:03.911181 30846 leader_election.cc:302] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:03.911315 31904 raft_consensus.cc:2781] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.911406 31904 raft_consensus.cc:686] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 LEADER]: Becoming Leader. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Running, Role: LEADER
I20240501 19:56:03.911568 31904 consensus_queue.cc:237] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:03.913057 30764 catalog_manager.cc:5484] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:04.043397 31994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.043582 31994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.046483 31994 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:04.119764 31171 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.119904 31302 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.120191 31303 raft_consensus.cc:1261] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.120440 31172 raft_consensus.cc:1261] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.120949 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.120889 31904 consensus_queue.cc:1035] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.121651 31987 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.121968 31987 consensus_queue.cc:1035] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.122438 31987 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.125340 30910 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.125732 31991 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.128943 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.133322 31991 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.448 ms
time per row: 0.0517238 ms
I20240501 19:56:04.187876 30765 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.192626 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 274800fe27c245139788f075eca1e812 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.192740 31283 tablet_service.cc:1467] Processing CreateTablet for tablet 274800fe27c245139788f075eca1e812 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.192934 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274800fe27c245139788f075eca1e812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.192974 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274800fe27c245139788f075eca1e812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.193156 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a882f8e74f2781f337f752516009 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.193383 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a882f8e74f2781f337f752516009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.194015 31152 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a882f8e74f2781f337f752516009 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.194259 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a882f8e74f2781f337f752516009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.195171 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a882f8e74f2781f337f752516009 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.195171 31019 tablet_service.cc:1467] Processing CreateTablet for tablet 6c061670c33145f389f4767a355537cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.195397 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a882f8e74f2781f337f752516009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.195489 31019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c061670c33145f389f4767a355537cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.196074 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 6c061670c33145f389f4767a355537cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.196221 31894 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:04.196293 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c061670c33145f389f4767a355537cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.196385 31895 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:04.197018 31897 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:04.197798 31897 tablet_bootstrap.cc:654] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.197842 30888 tablet_service.cc:1467] Processing CreateTablet for tablet 6c061670c33145f389f4767a355537cf (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.198081 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c061670c33145f389f4767a355537cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.198338 31894 tablet_bootstrap.cc:654] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.198499 30890 tablet_service.cc:1467] Processing CreateTablet for tablet 274800fe27c245139788f075eca1e812 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.198668 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274800fe27c245139788f075eca1e812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.199597 31894 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:04.199681 31894 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:04.200079 31893 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:04.200089 31894 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.200177 31894 raft_consensus.cc:374] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.200198 31894 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.200299 31894 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.200484 31895 tablet_bootstrap.cc:654] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.200538 31894 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:04.200647 31894 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:04.200891 31893 tablet_bootstrap.cc:654] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.201452 31894 tablet_bootstrap.cc:654] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.201594 31895 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:04.201673 31895 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:04.202034 31895 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.202129 31895 raft_consensus.cc:374] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.202168 31895 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.202303 31895 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.202541 31895 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.202745 31895 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:04.203063 31897 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:04.203132 31897 ts_tablet_manager.cc:1367] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:04.203506 31897 raft_consensus.cc:348] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.203586 31895 tablet_bootstrap.cc:654] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.203613 31897 raft_consensus.cc:374] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.203650 31897 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.203773 31897 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.203943 31894 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:04.204002 31894 ts_tablet_manager.cc:1367] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:04.204022 31897 ts_tablet_manager.cc:1398] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.204223 31893 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:04.204304 31893 ts_tablet_manager.cc:1367] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:04.204762 31893 raft_consensus.cc:348] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.204919 31893 raft_consensus.cc:374] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.204975 31893 raft_consensus.cc:729] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.205092 31893 consensus_queue.cc:260] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.205384 31893 ts_tablet_manager.cc:1398] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:04.205569 31893 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:04.205264 31894 raft_consensus.cc:348] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.206070 31894 raft_consensus.cc:374] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.206108 31894 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.206233 31894 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.206470 31893 tablet_bootstrap.cc:654] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.206539 31894 ts_tablet_manager.cc:1398] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:04.206806 31894 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:04.207453 31894 tablet_bootstrap.cc:654] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.207814 31893 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:04.207908 31893 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:04.208359 31894 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:04.208321 31893 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.208431 31894 ts_tablet_manager.cc:1367] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:04.208452 31893 raft_consensus.cc:374] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.208482 31893 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.208596 31893 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.208834 31893 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.208815 31894 raft_consensus.cc:348] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.208894 31894 raft_consensus.cc:374] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.208954 31894 raft_consensus.cc:729] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.209050 31894 consensus_queue.cc:260] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.209283 31894 ts_tablet_manager.cc:1398] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:04.209430 31895 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:04.209508 31895 ts_tablet_manager.cc:1367] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:04.209904 31895 raft_consensus.cc:348] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.210008 31895 raft_consensus.cc:374] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.210048 31895 raft_consensus.cc:729] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.210176 31895 consensus_queue.cc:260] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.210403 31895 ts_tablet_manager.cc:1398] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.210542 31895 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:04.211293 31895 tablet_bootstrap.cc:654] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.212323 31895 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:04.212390 31895 ts_tablet_manager.cc:1367] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:04.212985 31895 raft_consensus.cc:348] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.213155 31895 raft_consensus.cc:374] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.213225 31895 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.213373 31895 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.213620 31895 ts_tablet_manager.cc:1398] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.408304 31949 raft_consensus.cc:482] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.408397 31949 raft_consensus.cc:504] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.408725 31949 leader_election.cc:288] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:04.409128 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:04.409114 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:04.409263 31302 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.409258 30910 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.409569 30977 leader_election.cc:302] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:04.409765 31949 raft_consensus.cc:2781] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.409818 31949 raft_consensus.cc:482] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.409847 31949 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.410576 31949 raft_consensus.cc:504] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.410876 31949 leader_election.cc:288] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:04.411130 31949 raft_consensus.cc:482] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.411150 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:04.411192 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:04.411253 30910 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.411216 31949 raft_consensus.cc:504] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.411296 31302 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.411561 31949 leader_election.cc:288] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.411813 30909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c061670c33145f389f4767a355537cf" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:04.411916 30909 raft_consensus.cc:2454] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.411933 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c061670c33145f389f4767a355537cf" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:04.412029 31303 raft_consensus.cc:2454] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.412182 30910 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.412190 30975 leader_election.cc:302] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:04.412359 31302 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.412447 30975 leader_election.cc:302] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:04.412475 31949 raft_consensus.cc:2781] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.412514 31949 raft_consensus.cc:482] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.412549 31949 raft_consensus.cc:3037] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.412736 32038 raft_consensus.cc:2781] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.412874 32038 raft_consensus.cc:686] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:04.413024 32038 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.413219 31949 raft_consensus.cc:504] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.413606 31949 leader_election.cc:288] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.413882 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c061670c33145f389f4767a355537cf" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:04.413983 30910 raft_consensus.cc:3037] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.414275 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c061670c33145f389f4767a355537cf" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:04.414362 31302 raft_consensus.cc:3037] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.414633 30910 raft_consensus.cc:2454] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.414914 30975 leader_election.cc:302] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:04.414980 31302 raft_consensus.cc:2454] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.415102 31949 raft_consensus.cc:2781] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.415180 31949 raft_consensus.cc:686] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:04.415292 31949 consensus_queue.cc:237] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:04.415462 30765 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:04.418120 30765 catalog_manager.cc:5484] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.526098 31949 raft_consensus.cc:482] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.526185 31949 raft_consensus.cc:504] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.526455 31949 leader_election.cc:288] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.526783 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:04.526795 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:04.526901 31302 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.526901 31171 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:04.527153 30977 leader_election.cc:302] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:04.527312 31949 raft_consensus.cc:2781] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.527366 31949 raft_consensus.cc:482] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.527391 31949 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.528041 31949 raft_consensus.cc:504] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.528312 31949 leader_election.cc:288] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.528577 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:04.528604 31302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:04.528676 31171 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.528687 31302 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.529362 31302 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.529431 31171 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:04.529634 30977 leader_election.cc:302] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:04.529820 31949 raft_consensus.cc:2781] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.529912 31949 raft_consensus.cc:686] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:04.530031 31949 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } }
I20240501 19:56:04.531338 30765 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:04.655949 32043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.656121 32043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.658987 32043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:04.733440 30910 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.733551 31302 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.733793 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.734041 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.734226 32071 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.734452 31949 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.734232 32038 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.734988 32039 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.736388 30909 raft_consensus.cc:1261] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.736780 32039 consensus_queue.cc:1035] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.737057 31301 raft_consensus.cc:1261] T 6c061670c33145f389f4767a355537cf P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.739707 32039 consensus_queue.cc:1035] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.488 ms
time per row: 0.0532441 ms
I20240501 19:56:04.801837 30759 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.806658 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae5913b9cc43c1a82b2d03646983c5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.806933 31152 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae5913b9cc43c1a82b2d03646983c5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.807051 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae5913b9cc43c1a82b2d03646983c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.807150 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae5913b9cc43c1a82b2d03646983c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.807085 31151 tablet_service.cc:1467] Processing CreateTablet for tablet 2edc89ebb6984377ab459e5e219a4fab (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.807195 30890 tablet_service.cc:1467] Processing CreateTablet for tablet 33ae5913b9cc43c1a82b2d03646983c5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.807382 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edc89ebb6984377ab459e5e219a4fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.807693 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ae5913b9cc43c1a82b2d03646983c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.809657 32097 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:04.809799 32098 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:04.810663 32096 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:04.810698 32098 tablet_bootstrap.cc:654] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.810639 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 2edc89ebb6984377ab459e5e219a4fab (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.810675 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 174478acaefc4b29a001dd1f900e591b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.810876 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edc89ebb6984377ab459e5e219a4fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.810894 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.811489 32096 tablet_bootstrap.cc:654] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.812518 32097 tablet_bootstrap.cc:654] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.812572 32096 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:04.812647 32096 ts_tablet_manager.cc:1367] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.813107 32096 raft_consensus.cc:348] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.813225 32096 raft_consensus.cc:374] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.813275 32096 raft_consensus.cc:729] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.813392 32096 consensus_queue.cc:260] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.813549 32097 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:04.813612 32097 ts_tablet_manager.cc:1367] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:04.813658 32096 ts_tablet_manager.cc:1398] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.813689 30890 tablet_service.cc:1467] Processing CreateTablet for tablet 2edc89ebb6984377ab459e5e219a4fab (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.813923 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edc89ebb6984377ab459e5e219a4fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.813968 32096 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:04.814018 32097 raft_consensus.cc:348] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.814092 32097 raft_consensus.cc:374] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.814121 32097 raft_consensus.cc:729] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.814218 32097 consensus_queue.cc:260] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.814445 32097 ts_tablet_manager.cc:1398] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.814440 31152 tablet_service.cc:1467] Processing CreateTablet for tablet 174478acaefc4b29a001dd1f900e591b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.814584 32097 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:04.814657 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.814683 32096 tablet_bootstrap.cc:654] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.815313 32097 tablet_bootstrap.cc:654] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.816130 32097 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:04.816208 32097 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.816315 32096 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:04.816381 32096 ts_tablet_manager.cc:1367] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.816712 32096 raft_consensus.cc:348] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.816571 32097 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.816803 32097 raft_consensus.cc:374] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.816803 32096 raft_consensus.cc:374] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.816836 32097 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.816836 32096 raft_consensus.cc:729] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.816977 32096 consensus_queue.cc:260] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.816992 32097 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.817250 32096 ts_tablet_manager.cc:1398] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.817278 32097 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:04.817382 32096 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:04.817613 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 174478acaefc4b29a001dd1f900e591b (DEFAULT_TABLE table=rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.817838 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.817919 32096 tablet_bootstrap.cc:654] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.818782 32096 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:04.818845 32096 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.819113 32096 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.819187 32096 raft_consensus.cc:374] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.819221 32096 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.819344 32096 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.819568 32096 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.820477 32101 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:04.821498 32101 tablet_bootstrap.cc:654] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.822778 32098 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:04.822849 32098 ts_tablet_manager.cc:1367] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 19:56:04.822899 32101 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:04.822968 32101 ts_tablet_manager.cc:1367] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:04.823326 32101 raft_consensus.cc:348] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.823459 32101 raft_consensus.cc:374] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.823428 32098 raft_consensus.cc:348] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.823523 32101 raft_consensus.cc:729] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.823562 32098 raft_consensus.cc:374] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.823617 32098 raft_consensus.cc:729] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.823650 32101 consensus_queue.cc:260] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.823740 32098 consensus_queue.cc:260] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.823935 32101 ts_tablet_manager.cc:1398] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.824029 32098 ts_tablet_manager.cc:1398] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.824084 32101 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:04.824206 32098 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:04.824843 32098 tablet_bootstrap.cc:654] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.824906 32101 tablet_bootstrap.cc:654] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.826159 32098 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:04.826232 32098 ts_tablet_manager.cc:1367] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.826512 32101 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:04.826606 32101 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:04.826560 32098 raft_consensus.cc:348] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.826668 32098 raft_consensus.cc:374] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.826700 32098 raft_consensus.cc:729] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.826820 32098 consensus_queue.cc:260] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.827080 32098 ts_tablet_manager.cc:1398] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.827044 32101 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.827136 32101 raft_consensus.cc:374] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.827169 32101 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.827276 32101 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.827536 32101 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.834391 31909 raft_consensus.cc:482] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.834462 31909 raft_consensus.cc:504] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.834744 31909 leader_election.cc:288] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:04.835041 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae5913b9cc43c1a82b2d03646983c5" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:04.835041 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae5913b9cc43c1a82b2d03646983c5" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09" is_pre_election: true
I20240501 19:56:04.835156 30910 raft_consensus.cc:2454] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:04.835156 31041 raft_consensus.cc:2454] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:04.835412 31106 leader_election.cc:302] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:04.835620 31909 raft_consensus.cc:2781] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.835666 31909 raft_consensus.cc:482] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.835692 31909 raft_consensus.cc:3037] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.836315 31909 raft_consensus.cc:504] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.836599 31909 leader_election.cc:288] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:04.836863 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae5913b9cc43c1a82b2d03646983c5" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:04.836966 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ae5913b9cc43c1a82b2d03646983c5" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:04.837019 31041 raft_consensus.cc:3037] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.837071 30910 raft_consensus.cc:3037] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.837805 31041 raft_consensus.cc:2454] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:04.837827 30910 raft_consensus.cc:2454] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:04.838099 31106 leader_election.cc:302] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:04.838245 31909 raft_consensus.cc:2781] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.838449 31909 raft_consensus.cc:686] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:04.838565 31909 consensus_queue.cc:237] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:04.839773 30762 catalog_manager.cc:5484] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.993568 32020 raft_consensus.cc:482] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.993688 32020 raft_consensus.cc:504] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.994081 32020 leader_election.cc:288] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.994436 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:04.994472 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:04.994573 31303 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:04.994593 31171 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 0.
I20240501 19:56:04.994868 30844 leader_election.cc:302] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:04.995021 32020 raft_consensus.cc:2781] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.995083 32020 raft_consensus.cc:482] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.995121 32020 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.995877 32020 raft_consensus.cc:504] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.996181 32020 leader_election.cc:288] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:04.996475 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:04.996496 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:04.996575 31303 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.996593 31171 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.997323 31303 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 1.
I20240501 19:56:04.997475 31171 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00bf2be91e8f4a1a8b6fe07c478bf543 in term 1.
I20240501 19:56:04.997567 30846 leader_election.cc:302] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:04.997743 32020 raft_consensus.cc:2781] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.997936 32020 raft_consensus.cc:686] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 LEADER]: Becoming Leader. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Running, Role: LEADER
I20240501 19:56:04.998068 32020 consensus_queue.cc:237] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:04.999568 30762 catalog_manager.cc:5484] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.012943 32072 raft_consensus.cc:482] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.013054 32072 raft_consensus.cc:504] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:05.013974 32072 leader_election.cc:288] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:05.017668 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:05.017705 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09" is_pre_election: true
I20240501 19:56:05.017771 31171 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 0.
I20240501 19:56:05.017829 31041 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 0.
I20240501 19:56:05.018087 31237 leader_election.cc:302] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.018291 32072 raft_consensus.cc:2781] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.018359 32072 raft_consensus.cc:482] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.018409 32072 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.019138 32072 raft_consensus.cc:504] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:05.019412 32072 leader_election.cc:288] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 1 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:05.019690 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:05.019809 31171 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.019784 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:05.019886 31041 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.020511 31171 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 1.
I20240501 19:56:05.020550 31041 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 1.
I20240501 19:56:05.020792 31237 leader_election.cc:302] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.020947 32072 raft_consensus.cc:2781] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.021047 32072 raft_consensus.cc:686] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Becoming Leader. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Running, Role: LEADER
I20240501 19:56:05.021334 32072 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:05.023419 30762 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5530341eedde430fbacb4c3ff815159c (127.30.1.68). New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:05.145754 32117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.145954 32117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.148766 32117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:05.223934 30910 raft_consensus.cc:1261] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.224274 31041 raft_consensus.cc:1261] T 33ae5913b9cc43c1a82b2d03646983c5 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.224854 31909 consensus_queue.cc:1035] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.229760 31303 raft_consensus.cc:1261] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.230921 32111 consensus_queue.cc:1035] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.232551 31040 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.232846 31171 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.233331 32089 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.236500 32109 consensus_queue.cc:1035] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.236784 31172 raft_consensus.cc:1261] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 00bf2be91e8f4a1a8b6fe07c478bf543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.237418 32072 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.241135 32020 consensus_queue.cc:1035] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.988 ms
time per row: 0.0574939 ms
I20240501 19:56:05.300897 30761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57064:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.305853 31021 tablet_service.cc:1467] Processing CreateTablet for tablet ac9033765edb40b1ab49d4766eddf1ff (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.305853 31152 tablet_service.cc:1467] Processing CreateTablet for tablet ac9033765edb40b1ab49d4766eddf1ff (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.306110 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.306110 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.306804 30888 tablet_service.cc:1467] Processing CreateTablet for tablet 4f614c0ec5274c31a99bff11fe40307f (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.306993 30888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f614c0ec5274c31a99bff11fe40307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.308074 32096 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:05.308246 30890 tablet_service.cc:1467] Processing CreateTablet for tablet ac9033765edb40b1ab49d4766eddf1ff (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.308267 31282 tablet_service.cc:1467] Processing CreateTablet for tablet 4f614c0ec5274c31a99bff11fe40307f (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.308475 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.308475 31282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f614c0ec5274c31a99bff11fe40307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.308950 32096 tablet_bootstrap.cc:654] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.308974 32098 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:05.309144 31283 tablet_service.cc:1467] Processing CreateTablet for tablet 0684929c231d45a086b6356a9e6922ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.309343 31283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684929c231d45a086b6356a9e6922ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.309523 32097 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:05.309769 32098 tablet_bootstrap.cc:654] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.310256 32101 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:05.310274 32097 tablet_bootstrap.cc:654] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.310911 32098 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:05.310978 32098 ts_tablet_manager.cc:1367] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.310983 32101 tablet_bootstrap.cc:654] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.311064 30889 tablet_service.cc:1467] Processing CreateTablet for tablet 0684929c231d45a086b6356a9e6922ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.311300 30889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684929c231d45a086b6356a9e6922ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.311309 32098 raft_consensus.cc:348] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.311447 32098 raft_consensus.cc:374] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.311478 32098 raft_consensus.cc:729] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.311597 32098 consensus_queue.cc:260] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.311908 32098 ts_tablet_manager.cc:1398] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.312120 32101 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:05.312189 32101 ts_tablet_manager.cc:1367] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:05.312655 32101 raft_consensus.cc:348] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.312742 32101 raft_consensus.cc:374] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.312773 32101 raft_consensus.cc:729] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.312888 32101 consensus_queue.cc:260] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.313197 32101 ts_tablet_manager.cc:1398] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.313277 32098 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:05.313351 32101 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c: Bootstrap starting.
I20240501 19:56:05.314123 32098 tablet_bootstrap.cc:654] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.314169 32101 tablet_bootstrap.cc:654] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.315279 32101 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c: No bootstrap required, opened a new log
I20240501 19:56:05.315359 32101 ts_tablet_manager.cc:1367] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:05.316207 32101 raft_consensus.cc:348] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.316300 32101 raft_consensus.cc:374] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.316330 32101 raft_consensus.cc:729] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.316361 32098 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:05.316435 32098 ts_tablet_manager.cc:1367] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.316427 32101 consensus_queue.cc:260] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.316445 31152 tablet_service.cc:1467] Processing CreateTablet for tablet 0684929c231d45a086b6356a9e6922ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.316644 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684929c231d45a086b6356a9e6922ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.316704 32101 ts_tablet_manager.cc:1398] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.316973 32098 raft_consensus.cc:348] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.317170 31021 tablet_service.cc:1467] Processing CreateTablet for tablet 4f614c0ec5274c31a99bff11fe40307f (DEFAULT_TABLE table=rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.317099 32098 raft_consensus.cc:374] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.317387 31021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f614c0ec5274c31a99bff11fe40307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.317413 32098 raft_consensus.cc:729] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.317569 32097 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:05.317569 32098 consensus_queue.cc:260] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.317637 32097 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 19:56:05.318024 32097 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.318130 32097 raft_consensus.cc:374] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.318161 32097 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.318280 32097 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.318418 32098 ts_tablet_manager.cc:1398] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.318523 32097 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.318552 32098 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:05.319775 32098 tablet_bootstrap.cc:654] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.321593 32097 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09: Bootstrap starting.
I20240501 19:56:05.322160 32096 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:05.322245 32096 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20240501 19:56:05.322443 32097 tablet_bootstrap.cc:654] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.322714 32096 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.322834 32096 raft_consensus.cc:374] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.322880 32096 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.323009 32096 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.323267 32096 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.323377 32097 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09: No bootstrap required, opened a new log
I20240501 19:56:05.323390 32096 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a: Bootstrap starting.
I20240501 19:56:05.323410 32098 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: No bootstrap required, opened a new log
I20240501 19:56:05.323449 32097 ts_tablet_manager.cc:1367] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:05.323488 32098 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:05.323856 32097 raft_consensus.cc:348] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.323856 32098 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.323940 32098 raft_consensus.cc:374] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.323940 32097 raft_consensus.cc:374] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.323971 32098 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.323971 32097 raft_consensus.cc:729] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.324043 32096 tablet_bootstrap.cc:654] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.324076 32097 consensus_queue.cc:260] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.324079 32098 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.324316 32097 ts_tablet_manager.cc:1398] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.324469 32098 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.325956 32096 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a: No bootstrap required, opened a new log
I20240501 19:56:05.326031 32096 ts_tablet_manager.cc:1367] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:05.326398 32096 raft_consensus.cc:348] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.326474 32096 raft_consensus.cc:374] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.326494 32096 raft_consensus.cc:729] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.326589 32096 consensus_queue.cc:260] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.326838 32096 ts_tablet_manager.cc:1398] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.383190 31949 raft_consensus.cc:482] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.383278 31949 raft_consensus.cc:504] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.383617 31949 leader_election.cc:288] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.383904 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f614c0ec5274c31a99bff11fe40307f" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:05.383916 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f614c0ec5274c31a99bff11fe40307f" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:05.384011 30910 raft_consensus.cc:2454] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:05.384037 31303 raft_consensus.cc:2454] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 0.
I20240501 19:56:05.384276 30975 leader_election.cc:302] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:05.384418 31949 raft_consensus.cc:2781] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.384474 31949 raft_consensus.cc:482] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.384508 31949 raft_consensus.cc:3037] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.385250 31949 raft_consensus.cc:504] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.385556 31949 leader_election.cc:288] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.385854 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f614c0ec5274c31a99bff11fe40307f" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:05.385850 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f614c0ec5274c31a99bff11fe40307f" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:05.385946 31303 raft_consensus.cc:3037] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.385972 30910 raft_consensus.cc:3037] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.386591 30910 raft_consensus.cc:2454] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:05.386801 31303 raft_consensus.cc:2454] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 1.
I20240501 19:56:05.386839 30975 leader_election.cc:302] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 17d51481decd4d06964f09c66f981e09; no voters:
I20240501 19:56:05.386969 31949 raft_consensus.cc:2781] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.387110 31949 raft_consensus.cc:686] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:05.387248 31949 consensus_queue.cc:237] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.388605 30759 catalog_manager.cc:5484] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:05.423157 32155 raft_consensus.cc:482] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.423250 32155 raft_consensus.cc:504] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.423632 32155 leader_election.cc:288] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.423954 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09" is_pre_election: true
I20240501 19:56:05.423961 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:05.424046 30910 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:05.424057 31040 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:05.424333 31107 leader_election.cc:302] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.424502 32155 raft_consensus.cc:2781] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.424556 32155 raft_consensus.cc:482] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.424593 32155 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.425359 32155 raft_consensus.cc:504] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.425665 32155 leader_election.cc:288] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.425980 31040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:05.426012 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:05.426095 30910 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.426095 31040 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.426921 31040 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:05.426921 30910 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:05.427182 31107 leader_election.cc:302] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.427325 32155 raft_consensus.cc:2781] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.427402 32155 raft_consensus.cc:686] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:05.427524 32155 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.428970 30759 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:05.487533 32155 raft_consensus.cc:482] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.487622 32155 raft_consensus.cc:504] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.487978 32155 leader_election.cc:288] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.488343 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684929c231d45a086b6356a9e6922ec" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" is_pre_election: true
I20240501 19:56:05.488487 30910 raft_consensus.cc:2454] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:05.488771 31106 leader_election.cc:302] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.488955 32155 raft_consensus.cc:2781] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.489014 32155 raft_consensus.cc:482] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.489041 32155 raft_consensus.cc:3037] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.489876 32155 raft_consensus.cc:504] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.490229 32155 leader_election.cc:288] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:05.490525 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684929c231d45a086b6356a9e6922ec" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
I20240501 19:56:05.490630 30910 raft_consensus.cc:3037] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.491503 30910 raft_consensus.cc:2454] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:05.491739 31106 leader_election.cc:302] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:05.491897 32155 raft_consensus.cc:2781] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.491971 32155 raft_consensus.cc:686] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:05.491933 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684929c231d45a086b6356a9e6922ec" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:05.491999 31301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684929c231d45a086b6356a9e6922ec" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:05.492058 31303 raft_consensus.cc:2454] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 0.
I20240501 19:56:05.492108 32155 consensus_queue.cc:237] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:05.493714 30759 catalog_manager.cc:5484] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:05.599565 32175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.599731 32175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.602625 32175 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:05.678395 30910 raft_consensus.cc:1261] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.678797 31303 raft_consensus.cc:3037] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.678838 32109 consensus_queue.cc:1035] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.679831 31303 raft_consensus.cc:1261] T 0684929c231d45a086b6356a9e6922ec P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.681536 31909 consensus_queue.cc:1035] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.682404 31040 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.683218 31301 raft_consensus.cc:1261] T 4f614c0ec5274c31a99bff11fe40307f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.683238 30909 raft_consensus.cc:1261] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.684863 32170 consensus_queue.cc:1035] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.685159 32170 consensus_queue.cc:1035] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.685387 32155 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.692518 30908 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.695559 32109 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.727 ms
time per row: 0.0598635 ms
I20240501 19:56:05.759073 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:45745
--local_ip_for_outbound_sockets=127.30.1.69
--tserver_master_addrs=127.30.1.126:46341
--webserver_port=36021
--webserver_interface=127.30.1.69
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 19:56:05.859710 32213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.859894 32213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:05.859987 32213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.863178 32213 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:05.863246 32213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:05.863340 32213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:56:05.866719 32213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:45745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=36021
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.32213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:05.867439 32213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:05.868340 32213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:05.876128 32221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:05.876598 32219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:05.877409 32218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:05.878096 32213 server_base.cc:1034] running on GCE node
I20240501 19:56:05.878391 32213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:05.878840 32213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:05.879976 32213 hybrid_clock.cc:648] HybridClock initialized: now 1714593365879962 us; error 25 us; skew 500 ppm
I20240501 19:56:05.881700 32213 webserver.cc:460] Webserver started at http://127.30.1.69:36021/ using document root <none> and password file <none>
I20240501 19:56:05.882203 32213 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:05.882267 32213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:05.885283 32213 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:05.886899 32227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.887810 32213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.887902 32213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3393eeb225a4430382811e9bef2948fe"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:56:05.888224 32213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:05.909390 32213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:05.909883 32213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:05.910014 32213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:05.910462 32213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:05.911208 32213 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:05.911263 32213 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.911310 32213 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:05.911362 32213 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:05.919485 32213 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:45745
I20240501 19:56:05.919517 32340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:45745 every 8 connection(s)
I20240501 19:56:05.920315 32213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:05.923794 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 32213
I20240501 19:56:05.924141 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:39065
--local_ip_for_outbound_sockets=127.30.1.70
--tserver_master_addrs=127.30.1.126:46341
--webserver_port=37691
--webserver_interface=127.30.1.70
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:56:05.928207 32341 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:56:05.928393 32341 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:05.928819 32341 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:56:05.929936 30761 ts_manager.cc:194] Re-registered known tserver with Master: 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745)
I20240501 19:56:05.930733 30761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:58493
W20240501 19:56:06.024762 32344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.024976 32344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.025034 32344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.027827 32344 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.027876 32344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.027940 32344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:56:06.031301 32344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:39065
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=37691
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.32344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.032037 32344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.033078 32344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.040972 32349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.041410 32350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.042150 32352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.042928 32344 server_base.cc:1034] running on GCE node
I20240501 19:56:06.043304 32344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.043807 32344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.044963 32344 hybrid_clock.cc:648] HybridClock initialized: now 1714593366044946 us; error 27 us; skew 500 ppm
I20240501 19:56:06.046715 32344 webserver.cc:460] Webserver started at http://127.30.1.70:37691/ using document root <none> and password file <none>
I20240501 19:56:06.047232 32344 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.047309 32344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.050429 32344 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:06.052260 32358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.053189 32344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.053287 32344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:56:06.053661 32344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.066804 32344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.067330 32344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.067468 32344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.067932 32344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.068748 32344 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.068799 32344 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.068853 32344 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.068884 32344 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.077914 32344 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:39065
I20240501 19:56:06.077937 32471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:39065 every 8 connection(s)
I20240501 19:56:06.078816 32344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:06.079113 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 32344
I20240501 19:56:06.079546 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:37429
--local_ip_for_outbound_sockets=127.30.1.71
--tserver_master_addrs=127.30.1.126:46341
--webserver_port=32975
--webserver_interface=127.30.1.71
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:56:06.087642 32472 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:56:06.087831 32472 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.088270 32472 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:56:06.089179 30761 ts_manager.cc:194] Re-registered known tserver with Master: 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065)
I20240501 19:56:06.089694 30761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:55993
W20240501 19:56:06.182974 32474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.183156 32474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.183213 32474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.186139 32474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.186215 32474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.186298 32474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:56:06.189554 32474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:37429
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=32975
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.32474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.190339 32474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.191288 32474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.198664 32480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.199306 32483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.199987 32481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.200712 32474 server_base.cc:1034] running on GCE node
I20240501 19:56:06.201053 32474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.201570 32474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.202744 32474 hybrid_clock.cc:648] HybridClock initialized: now 1714593366202726 us; error 31 us; skew 500 ppm
I20240501 19:56:06.204430 32474 webserver.cc:460] Webserver started at http://127.30.1.71:32975/ using document root <none> and password file <none>
I20240501 19:56:06.205000 32474 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.205086 32474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.208194 32474 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.209911 32489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.210875 32474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.210995 32474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "af53e6b6c5e549dda69f656f9a755c5b"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:56:06.211386 32474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.226851 32474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.227305 32474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.227437 32474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.227859 32474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.228574 32474 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.228617 32474 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.228682 32474 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.228716 32474 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.238029 32474 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:37429
I20240501 19:56:06.238327 32602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:37429 every 8 connection(s)
I20240501 19:56:06.238982 32474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:06.244972 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 32474
I20240501 19:56:06.245394 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.72:45409
--local_ip_for_outbound_sockets=127.30.1.72
--tserver_master_addrs=127.30.1.126:46341
--webserver_port=40187
--webserver_interface=127.30.1.72
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 19:56:06.246819 32603 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:56:06.247048 32603 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.247594 32603 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:56:06.248466 30761 ts_manager.cc:194] Re-registered known tserver with Master: af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429)
I20240501 19:56:06.249023 30761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:40179
W20240501 19:56:06.347854 32606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.348021 32606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:06.348083 32606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.350953 32606 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:06.351016 32606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:06.351104 32606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.72
I20240501 19:56:06.354714 32606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.72:45409
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.1.72
--webserver_port=40187
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.32606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.72
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:06.355410 32606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:06.356364 32606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.363672 32611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.364413 32612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.365130 32614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.365787 32606 server_base.cc:1034] running on GCE node
I20240501 19:56:06.366055 32606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:06.366453 32606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:06.367586 32606 hybrid_clock.cc:648] HybridClock initialized: now 1714593366367573 us; error 24 us; skew 500 ppm
I20240501 19:56:06.369191 32606 webserver.cc:460] Webserver started at http://127.30.1.72:40187/ using document root <none> and password file <none>
I20240501 19:56:06.369659 32606 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.369740 32606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.372792 32606 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:06.374490 32620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.375389 32606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.375471 32606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a159fd4137c34800b2f1d142e2272705"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-htmr"
I20240501 19:56:06.375803 32606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.395625 32606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.396103 32606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:06.396227 32606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.396678 32606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:06.397454 32606 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:06.397538 32606 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.397591 32606 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:06.397607 32606 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.406002 32606 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.72:45409
I20240501 19:56:06.406052 32733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.72:45409 every 8 connection(s)
I20240501 19:56:06.406916 32606 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:06.410626 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 32606
I20240501 19:56:06.415509 32734 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:56:06.415695 32734 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:06.416169 32734 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:56:06.417153 30761 ts_manager.cc:194] Re-registered known tserver with Master: a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:56:06.417963 30761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.72:37661
I20240501 19:56:06.585179 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:06.585695 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.586953 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:06.589098 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.607510 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:06.608284 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.609692 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:06.615288 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:06.774813 30762 ts_manager.cc:292] Set tserver state for 00bf2be91e8f4a1a8b6fe07c478bf543 to MAINTENANCE_MODE
W20240501 19:56:06.896641 330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.896831 330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.899842 330 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:06.932416 32341 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:06.980865 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:06.987512 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:06.992012 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.002161 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.010545 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:07.016423 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.017212 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:07.018195 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:07.025947 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.034409 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.042694 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.047744 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.059697 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.068105 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:07.091607 32472 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:07.100260 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.100243 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.216975 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:07.220041 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.227726 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:07.241147 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.244143 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:07.250725 32603 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:07.252554 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.272432 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.272987 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.400733 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.401815 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:56:07.406646 425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:07.406874 425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 19:56:07.410835 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:56:07.411070 425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:07.417918 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.418645 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:07.419628 32734 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:07.421427 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.435428 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:07.442481 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.479434 31041 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:07.480614 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:07.480705 30908 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:07.481019 32170 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:07.481402 500 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:07.482921 32170 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } }
I20240501 19:56:07.483350 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } }
I20240501 19:56:07.484179 30908 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } }
I20240501 19:56:07.505226 30762 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } }
I20240501 19:56:07.515895 31041 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:07.518040 30908 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:07.518545 500 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:07.518905 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:07.519284 500 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:07.520354 500 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:07.522076 30908 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:07.522543 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
W20240501 19:56:07.528483 30974 consensus_peers.cc:485] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 -> Peer 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065): Couldn't send request to peer 1cd4016c2d8f4feb9fb1fd5a8be29fca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33c7d50279364b53953d0a0d45edfb9a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:07.530871 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.532073 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.538873 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.540297 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.556644 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.602340 30759 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 reported cstate change: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:07.603201 30754 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 33c7d50279364b53953d0a0d45edfb9a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:07.605059 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:07.605135 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:07.606151 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Runtime error: ksck discovered errors
W20240501 19:56:07.753381 546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:07.753566 546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:07.756309 546 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:07.834148 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:07.834578 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.836050 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:07.842518 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:07.846005 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.847148 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:07.865445 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.875332 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:07.930584 594 ts_tablet_manager.cc:902] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:07.931376 594 tablet_copy_client.cc:308] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:07.935439 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 33c7d50279364b53953d0a0d45edfb9a from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca ({username='slave'} at 127.30.1.70:37359)
I20240501 19:56:07.935578 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 33c7d50279364b53953d0a0d45edfb9a from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca at {username='slave'} at 127.30.1.70:37359: session id = 1cd4016c2d8f4feb9fb1fd5a8be29fca-33c7d50279364b53953d0a0d45edfb9a
I20240501 19:56:07.936972 31061 tablet_copy_source_session.cc:205] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:07.940016 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.945226 594 tablet_copy_client.cc:783] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:07.945401 594 tablet_copy_client.cc:647] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:07.946836 598 tablet_copy_client.cc:1002] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:08.024854 588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:08.025115 588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:08.028069 588 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:08.111090 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:08.112038 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:08.114099 30890 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:08.122023 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:08.129005 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:08.129973 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:08.141592 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:08.148371 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240501 19:56:08.181099 328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 30828
W20240501 19:56:08.252842 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:08.253892 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.354079 30975 consensus_peers.cc:485] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.354719 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.387504 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:08.388207 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.479725 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.480752 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.631660 31106 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.683926 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:08.737900 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:09.106110 670 raft_consensus.cc:482] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.106220 670 raft_consensus.cc:504] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.106562 670 leader_election.cc:288] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.106853 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
W20240501 19:56:09.107416 31106 leader_election.cc:334] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.107474 31106 leader_election.cc:302] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac3d66d1d57441db81def86faef0eb4a; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c
I20240501 19:56:09.107656 670 raft_consensus.cc:2726] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:09.123405 32089 raft_consensus.cc:482] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.123499 32089 raft_consensus.cc:504] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.123833 32089 leader_election.cc:288] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.124169 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
W20240501 19:56:09.124473 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:09.125197 31237 leader_election.cc:334] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.125259 31237 leader_election.cc:302] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543, ac3d66d1d57441db81def86faef0eb4a
I20240501 19:56:09.125383 32089 raft_consensus.cc:2726] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:09.215644 32109 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.215727 32109 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.216056 32109 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.216378 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
W20240501 19:56:09.216944 31106 leader_election.cc:334] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.217007 31106 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac3d66d1d57441db81def86faef0eb4a; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543, 5530341eedde430fbacb4c3ff815159c
I20240501 19:56:09.217175 32109 raft_consensus.cc:2726] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:09.447296 594 tablet_copy_client.cc:523] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:09.448959 594 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap starting.
I20240501 19:56:09.454351 594 log.cc:824] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:09.472164 594 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:09.472410 594 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap complete.
I20240501 19:56:09.472638 594 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20240501 19:56:09.474385 594 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:09.474571 594 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Initialized, Role: LEARNER
I20240501 19:56:09.474864 594 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:09.475524 594 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:09.476063 32426 raft_consensus.cc:1203] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:09.476156 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-33c7d50279364b53953d0a0d45edfb9a received from {username='slave'} at 127.30.1.70:37359
I20240501 19:56:09.476248 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-33c7d50279364b53953d0a0d45edfb9a on tablet 33c7d50279364b53953d0a0d45edfb9a with peer 1cd4016c2d8f4feb9fb1fd5a8be29fca
I20240501 19:56:09.581920 32089 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.582026 32089 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.582368 32089 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.582674 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:09.582798 31171 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 1.
I20240501 19:56:09.583012 31237 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:09.583205 32089 raft_consensus.cc:2781] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:09.583269 32089 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.583319 32089 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:09.583562 31237 leader_election.cc:334] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.584301 32089 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.584622 32089 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.584887 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:09.585026 31171 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:09.585503 31237 leader_election.cc:334] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.585871 31171 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 2.
I20240501 19:56:09.586117 31237 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543
I20240501 19:56:09.586248 32089 raft_consensus.cc:2781] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.586400 32089 raft_consensus.cc:686] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Becoming Leader. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Running, Role: LEADER
I20240501 19:56:09.586524 32089 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.588059 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: term changed from 1 to 2, leader changed from 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) to 5530341eedde430fbacb4c3ff815159c (127.30.1.68). New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:09.623363 670 raft_consensus.cc:482] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.623446 670 raft_consensus.cc:504] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.623785 670 leader_election.cc:288] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.624114 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c" is_pre_election: true
I20240501 19:56:09.624234 31303 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 1.
I20240501 19:56:09.624502 31108 leader_election.cc:302] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:09.624667 670 raft_consensus.cc:2781] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:09.624712 670 raft_consensus.cc:482] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
W20240501 19:56:09.624747 31106 leader_election.cc:334] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.624759 670 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.625561 670 raft_consensus.cc:504] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.625844 670 leader_election.cc:288] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.626120 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2edc89ebb6984377ab459e5e219a4fab" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:09.626235 31303 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:09.626814 31106 leader_election.cc:334] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.627135 31303 raft_consensus.cc:2454] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 2.
I20240501 19:56:09.627378 31108 leader_election.cc:302] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543
I20240501 19:56:09.627557 670 raft_consensus.cc:2781] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.627663 670 raft_consensus.cc:686] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [term 2 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:09.627785 670 consensus_queue.cc:237] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.629108 30761 catalog_manager.cc:5484] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 1 to 2, leader changed from 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 2 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:09.656582 32089 raft_consensus.cc:482] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.656677 32089 raft_consensus.cc:504] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.657014 32089 leader_election.cc:288] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.657385 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a" is_pre_election: true
I20240501 19:56:09.657518 31171 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 1.
I20240501 19:56:09.657996 31237 leader_election.cc:302] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:09.658164 32089 raft_consensus.cc:2781] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 19:56:09.658174 31237 leader_election.cc:334] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.658234 32089 raft_consensus.cc:482] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00bf2be91e8f4a1a8b6fe07c478bf543)
I20240501 19:56:09.658550 32089 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:09.659317 32089 raft_consensus.cc:504] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.659615 32089 leader_election.cc:288] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:09.659889 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2ccd7f3fea4473497944de737911828" candidate_uuid: "5530341eedde430fbacb4c3ff815159c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:09.659996 31171 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:09.660583 31237 leader_election.cc:334] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:09.660849 31171 raft_consensus.cc:2454] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5530341eedde430fbacb4c3ff815159c in term 2.
I20240501 19:56:09.661101 31237 leader_election.cc:302] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters: 00bf2be91e8f4a1a8b6fe07c478bf543
I20240501 19:56:09.661242 32089 raft_consensus.cc:2781] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:09.661332 32089 raft_consensus.cc:686] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Becoming Leader. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Running, Role: LEADER
I20240501 19:56:09.661459 32089 consensus_queue.cc:237] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:09.662873 30761 catalog_manager.cc:5484] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c reported cstate change: term changed from 1 to 2, leader changed from 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) to 5530341eedde430fbacb4c3ff815159c (127.30.1.68). New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:09.867839 703 raft_consensus.cc:1050] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca to VOTER
I20240501 19:56:09.868338 703 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:09.869634 32426 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:09.869761 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:09.870146 703 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:09.870266 30975 consensus_peers.cc:485] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:09.870306 704 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:09.872658 703 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.873188 32426 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.874532 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.875115 30761 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.877929 31041 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:09.878904 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:09.878904 32426 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:09.879284 706 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:09.879416 645 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:09.880918 706 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.881333 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.881444 32426 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.883369 30765 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:09.883705 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33c7d50279364b53953d0a0d45edfb9a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 19:56:09.886303 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:09.887002 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 33c7d50279364b53953d0a0d45edfb9a: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:10.058156 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:10.162668 31171 raft_consensus.cc:1261] T b2ccd7f3fea4473497944de737911828 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:10.163040 32089 consensus_queue.cc:1035] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:10.178020 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:10.178418 717 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:10.181376 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:10.181656 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:10.205186 31303 raft_consensus.cc:1261] T 2edc89ebb6984377ab459e5e219a4fab P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:10.205569 670 consensus_queue.cc:1035] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:10.217993 31171 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:10.219199 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:10.219588 720 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:10.219743 31106 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:10.221128 670 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } }
I20240501 19:56:10.221515 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } }
I20240501 19:56:10.223520 30759 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:10.226471 31171 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:10.228119 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:10.228471 737 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:10.228732 31106 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:10.230010 720 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:10.231334 30753 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dbe3171686254a27a3bbe08786fa7503 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:10.231806 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:10.232039 30759 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 4 to 5, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:10.234658 31107 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbe3171686254a27a3bbe08786fa7503. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:10.517498 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:10.846937 756 ts_tablet_manager.cc:902] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:10.847621 756 tablet_copy_client.cc:308] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:36463
I20240501 19:56:10.851374 31192 tablet_copy_service.cc:140] P ac3d66d1d57441db81def86faef0eb4a: Received BeginTabletCopySession request for tablet dbe3171686254a27a3bbe08786fa7503 from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:49045)
I20240501 19:56:10.851485 31192 tablet_copy_service.cc:161] P ac3d66d1d57441db81def86faef0eb4a: Beginning new tablet copy session on tablet dbe3171686254a27a3bbe08786fa7503 from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:49045: session id = a159fd4137c34800b2f1d142e2272705-dbe3171686254a27a3bbe08786fa7503
I20240501 19:56:10.852556 31192 tablet_copy_source_session.cc:205] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:10.855506 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe3171686254a27a3bbe08786fa7503. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:56:10.859068 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:10.860541 756 tablet_copy_client.cc:783] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:10.860720 756 tablet_copy_client.cc:647] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:10.861969 760 tablet_copy_client.cc:1002] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:10.940021 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:11.095805 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 33c7d50279364b53953d0a0d45edfb9a: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:11.103837 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:11.120185 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:11.185892 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:11.212563 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:12.163592 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 33c7d50279364b53953d0a0d45edfb9a: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:12.362360 756 tablet_copy_client.cc:523] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:12.364151 756 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:12.369558 756 log.cc:824] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:12.387933 756 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:12.388180 756 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:12.388414 756 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20240501 19:56:12.390193 756 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:12.390372 756 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:12.390659 756 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:12.391511 756 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 19:56:12.392102 31192 tablet_copy_service.cc:342] P ac3d66d1d57441db81def86faef0eb4a: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-dbe3171686254a27a3bbe08786fa7503 received from {username='slave'} at 127.30.1.72:49045
I20240501 19:56:12.392200 31192 tablet_copy_service.cc:433] P ac3d66d1d57441db81def86faef0eb4a: ending tablet copy session a159fd4137c34800b2f1d142e2272705-dbe3171686254a27a3bbe08786fa7503 on tablet dbe3171686254a27a3bbe08786fa7503 with peer a159fd4137c34800b2f1d142e2272705
I20240501 19:56:12.395614 32688 raft_consensus.cc:1203] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 19:56:12.594347 31106 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:12.608063 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:12.717881 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:12.748965 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:12.775262 819 raft_consensus.cc:1050] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:12.775601 819 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:12.776865 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:12.777299 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:56:12.777318 31106 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:12.777340 818 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:12.777647 818 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:12.779763 821 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.780256 32688 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.781482 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.781983 30761 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 5 to 6, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:12.785286 31172 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:12.786177 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:12.786299 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:12.786643 788 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:12.786742 819 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:12.788206 819 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.788563 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.788627 32688 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:12.790092 30753 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbe3171686254a27a3bbe08786fa7503 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:12.790426 30765 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:13.115048 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } }
I20240501 19:56:13.116390 31172 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 19:56:13.116590 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:13.116807 842 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:13.118474 722 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:13.118907 31172 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:13.121302 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:13.123279 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:13.124397 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:13.126310 31172 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:13.126940 844 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:13.127338 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:13.128641 722 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:13.129138 31172 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
W20240501 19:56:13.130524 31238 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb93992159a3461b945d31da3c9bb43f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:13.131356 30755 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bb93992159a3461b945d31da3c9bb43f with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:13.131510 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 5 to 6, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
W20240501 19:56:13.327636 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240501 19:56:13.328227 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:13.405288 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbe3171686254a27a3bbe08786fa7503: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:13.437436 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 19:56:13.438050 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:13.512926 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:13.516959 857 ts_tablet_manager.cc:902] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:13.517356 857 tablet_copy_client.cc:308] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:37085
I20240501 19:56:13.520483 31323 tablet_copy_service.cc:140] P 5530341eedde430fbacb4c3ff815159c: Received BeginTabletCopySession request for tablet bb93992159a3461b945d31da3c9bb43f from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:42725)
I20240501 19:56:13.520593 31323 tablet_copy_service.cc:161] P 5530341eedde430fbacb4c3ff815159c: Beginning new tablet copy session on tablet bb93992159a3461b945d31da3c9bb43f from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:42725: session id = a159fd4137c34800b2f1d142e2272705-bb93992159a3461b945d31da3c9bb43f
I20240501 19:56:13.521960 31323 tablet_copy_source_session.cc:205] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:13.522961 857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.524587 857 tablet_copy_client.cc:783] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:13.524734 857 tablet_copy_client.cc:647] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:13.525777 861 tablet_copy_client.cc:1002] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:13.549173 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:13.672210 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:13.677675 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:56:14.410429 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20240501 19:56:14.983080 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240501 19:56:14.983706 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbe3171686254a27a3bbe08786fa7503: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:15.026041 857 tablet_copy_client.cc:523] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:15.027324 857 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:15.044457 857 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:15.044698 857 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:15.044847 857 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 19:56:15.045305 857 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:15.045401 857 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:15.045492 857 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:15.045778 857 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240501 19:56:15.046218 31323 tablet_copy_service.cc:342] P 5530341eedde430fbacb4c3ff815159c: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-bb93992159a3461b945d31da3c9bb43f received from {username='slave'} at 127.30.1.72:42725
I20240501 19:56:15.046306 31323 tablet_copy_service.cc:433] P 5530341eedde430fbacb4c3ff815159c: ending tablet copy session a159fd4137c34800b2f1d142e2272705-bb93992159a3461b945d31da3c9bb43f on tablet bb93992159a3461b945d31da3c9bb43f with peer a159fd4137c34800b2f1d142e2272705
W20240501 19:56:15.145251 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:15.158926 32688 raft_consensus.cc:1203] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240501 19:56:15.365957 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:15.561278 887 raft_consensus.cc:1050] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:15.561661 887 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:15.563042 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:15.563381 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 19:56:15.563429 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:56:15.563443 925 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:15.563558 31237 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:15.563746 924 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:15.565371 925 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.565690 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.567154 31171 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.567708 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 6 to 7, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:15.570742 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:15.571653 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:15.571653 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:15.572057 924 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:15.572183 925 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:15.573590 922 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.574174 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.574985 30755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb93992159a3461b945d31da3c9bb43f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:15.575382 31171 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:15.575799 30765 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 7 to 8, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:15.703909 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:15.754297 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:56:15.800105 31171 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } }
I20240501 19:56:15.801285 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:15.801642 946 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:15.801806 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:15.803072 821 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:15.803504 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:15.805851 30759 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:15.808511 31171 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:15.809599 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:15.810081 948 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:15.811221 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:15.811561 948 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:15.813215 30753 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac9033765edb40b1ab49d4766eddf1ff with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:15.813288 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:15.815234 30759 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 4 to 5, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:15.816284 31105 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065): Couldn't send request to peer 1cd4016c2d8f4feb9fb1fd5a8be29fca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac9033765edb40b1ab49d4766eddf1ff. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:15.844069 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:15.900766 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:16.158015 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:16.185544 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb93992159a3461b945d31da3c9bb43f: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:16.272989 968 ts_tablet_manager.cc:902] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Initiating tablet copy from peer ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:16.273458 968 tablet_copy_client.cc:308] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:36463
I20240501 19:56:16.276486 31192 tablet_copy_service.cc:140] P ac3d66d1d57441db81def86faef0eb4a: Received BeginTabletCopySession request for tablet ac9033765edb40b1ab49d4766eddf1ff from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca ({username='slave'} at 127.30.1.70:50703)
I20240501 19:56:16.276573 31192 tablet_copy_service.cc:161] P ac3d66d1d57441db81def86faef0eb4a: Beginning new tablet copy session on tablet ac9033765edb40b1ab49d4766eddf1ff from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca at {username='slave'} at 127.30.1.70:50703: session id = 1cd4016c2d8f4feb9fb1fd5a8be29fca-ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:16.277454 31192 tablet_copy_source_session.cc:205] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:16.278221 968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:16.280176 968 tablet_copy_client.cc:783] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:16.280375 968 tablet_copy_client.cc:647] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:16.281224 972 tablet_copy_client.cc:1002] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:16.327772 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:17.676012 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:56:17.781471 968 tablet_copy_client.cc:523] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:17.782646 968 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap starting.
W20240501 19:56:17.786252 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb93992159a3461b945d31da3c9bb43f: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:17.800300 968 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:17.800519 968 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap complete.
I20240501 19:56:17.800678 968 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 19:56:17.801339 968 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:17.801491 968 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Initialized, Role: LEARNER
I20240501 19:56:17.801651 968 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:17.802290 968 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 19:56:17.802774 31192 tablet_copy_service.cc:342] P ac3d66d1d57441db81def86faef0eb4a: Request end of tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-ac9033765edb40b1ab49d4766eddf1ff received from {username='slave'} at 127.30.1.70:50703
I20240501 19:56:17.802863 31192 tablet_copy_service.cc:433] P ac3d66d1d57441db81def86faef0eb4a: ending tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-ac9033765edb40b1ab49d4766eddf1ff on tablet ac9033765edb40b1ab49d4766eddf1ff with peer 1cd4016c2d8f4feb9fb1fd5a8be29fca
I20240501 19:56:17.854066 32426 raft_consensus.cc:1203] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 19:56:18.071460 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:18.187520 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:18.189841 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:18.283839 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:56:18.291935 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:18.369195 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 19:56:18.370358 1041 raft_consensus.cc:1050] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: attempting to promote NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca to VOTER
I20240501 19:56:18.370826 1041 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:18.372159 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:18.372583 1044 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:18.372833 31106 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:18.372865 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:18.373435 1044 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:18.375149 1043 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.375566 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.376694 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.377207 30761 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 5 to 6, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:18.380311 31171 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:18.381305 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:18.381400 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:18.382027 1041 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:18.382130 1044 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:18.383601 1006 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.383971 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.385114 30753 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac9033765edb40b1ab49d4766eddf1ff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:18.385195 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:18.386129 30759 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:18.423611 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:18.531179 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 19:56:18.567242 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:56:18.699491 31041 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } }
I20240501 19:56:18.700635 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:18.701000 1066 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:18.701148 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:18.702319 704 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:18.702687 31303 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } }
I20240501 19:56:18.704715 30761 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:18.707499 31041 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:18.709789 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:18.710114 704 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:18.710700 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:18.711519 1073 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:18.711967 31303 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:18.712941 30754 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 274800fe27c245139788f075eca1e812 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:18.714165 30761 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 4 to 5, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:18.714982 30974 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429): Couldn't send request to peer af53e6b6c5e549dda69f656f9a755c5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 274800fe27c245139788f075eca1e812. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:18.926098 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:19.003120 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ac9033765edb40b1ab49d4766eddf1ff: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:19.181391 1086 ts_tablet_manager.cc:902] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:19.182200 1086 tablet_copy_client.cc:308] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:19.185623 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 274800fe27c245139788f075eca1e812 from peer af53e6b6c5e549dda69f656f9a755c5b ({username='slave'} at 127.30.1.71:54853)
I20240501 19:56:19.185707 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 274800fe27c245139788f075eca1e812 from peer af53e6b6c5e549dda69f656f9a755c5b at {username='slave'} at 127.30.1.71:54853: session id = af53e6b6c5e549dda69f656f9a755c5b-274800fe27c245139788f075eca1e812
I20240501 19:56:19.186468 31061 tablet_copy_source_session.cc:205] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:19.189337 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274800fe27c245139788f075eca1e812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.194221 1086 tablet_copy_client.cc:783] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:19.194401 1086 tablet_copy_client.cc:647] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:19.195461 1090 tablet_copy_client.cc:1002] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:19.729749 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 19:56:19.944512 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:56:20.532409 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:20.534983 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:20.569316 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 19:56:20.570024 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ac9033765edb40b1ab49d4766eddf1ff: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:20.577113 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:20.640872 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:56:20.695845 1086 tablet_copy_client.cc:523] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:20.697806 1086 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap starting.
I20240501 19:56:20.702857 1086 log.cc:824] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:20.722148 1086 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:20.722375 1086 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap complete.
I20240501 19:56:20.722577 1086 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240501 19:56:20.724200 1086 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:20.724380 1086 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Initialized, Role: LEARNER
I20240501 19:56:20.724625 1086 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:20.725286 1086 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:20.725901 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session af53e6b6c5e549dda69f656f9a755c5b-274800fe27c245139788f075eca1e812 received from {username='slave'} at 127.30.1.71:54853
I20240501 19:56:20.725970 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session af53e6b6c5e549dda69f656f9a755c5b-274800fe27c245139788f075eca1e812 on tablet 274800fe27c245139788f075eca1e812 with peer af53e6b6c5e549dda69f656f9a755c5b
I20240501 19:56:20.956193 32540 raft_consensus.cc:1203] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 19:56:20.994899 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:21.265642 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:21.314496 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:56:21.354288 1160 raft_consensus.cc:1050] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER af53e6b6c5e549dda69f656f9a755c5b to VOTER
I20240501 19:56:21.354648 1160 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:21.355871 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:21.355978 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:21.356319 1107 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:21.356441 30975 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:21.356469 1163 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:21.358994 1107 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.359500 32540 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.360625 31303 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.361335 30765 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 5 to 6, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:21.365165 31041 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:21.366206 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:21.366206 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:21.366569 1161 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:21.367065 1161 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:21.367894 1160 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.368309 31303 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.368875 32540 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:21.369604 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 274800fe27c245139788f075eca1e812 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:21.370729 30759 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 6 to 7, VOTER 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
W20240501 19:56:21.571480 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 274800fe27c245139788f075eca1e812: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:21.635280 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:21.636834 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.639492 31283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.645345 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.649364 31152 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:21.658725 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.659087 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.743135 31021 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:21.743515 32406 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.746438 32668 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.753476 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.756728 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:21.761881 31283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:21.764295 31152 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:21.811394 31303 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:21.812870 31171 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:21.812875 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:21.813271 1261 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:21.813513 925 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:21.815441 1260 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:21.815871 31171 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:21.817839 31041 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:21.818545 30768 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
W20240501 19:56:21.819803 31236 consensus_peers.cc:485] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c -> Peer 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745): Couldn't send request to peer 3393eeb225a4430382811e9bef2948fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 174478acaefc4b29a001dd1f900e591b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:21.828837 31041 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:21.830294 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:21.830401 32426 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 19:56:21.830587 30974 consensus_peers.cc:485] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 -> Peer af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429): Couldn't send request to peer af53e6b6c5e549dda69f656f9a755c5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33c7d50279364b53953d0a0d45edfb9a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:21.830684 1163 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:21.830835 1160 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:21.832481 1160 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:21.833029 32426 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:21.835176 30761 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 7 to 8, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:21.836066 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:21.843760 31041 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:21.845137 31303 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:21.845137 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:21.845255 30974 consensus_peers.cc:485] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 -> Peer 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065): Couldn't send request to peer 1cd4016c2d8f4feb9fb1fd5a8be29fca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f84a882f8e74f2781f337f752516009. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:21.845526 1161 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:21.845724 1107 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:21.847476 1160 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:21.847852 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:21.848219 31303 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:21.850127 30761 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index -1 to 4, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:22.316475 1327 ts_tablet_manager.cc:902] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:22.316880 1327 tablet_copy_client.cc:308] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:22.317390 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 33c7d50279364b53953d0a0d45edfb9a from peer af53e6b6c5e549dda69f656f9a755c5b ({username='slave'} at 127.30.1.71:54853)
I20240501 19:56:22.317473 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 33c7d50279364b53953d0a0d45edfb9a from peer af53e6b6c5e549dda69f656f9a755c5b at {username='slave'} at 127.30.1.71:54853: session id = af53e6b6c5e549dda69f656f9a755c5b-33c7d50279364b53953d0a0d45edfb9a
I20240501 19:56:22.318156 31061 tablet_copy_source_session.cc:205] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:22.319211 1327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.321132 1327 tablet_copy_client.cc:783] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:22.321270 1327 tablet_copy_client.cc:647] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:22.322158 1331 tablet_copy_client.cc:1002] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:22.381875 1334 ts_tablet_manager.cc:902] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:22.382529 1334 tablet_copy_client.cc:308] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:22.383329 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 8f84a882f8e74f2781f337f752516009 from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca ({username='slave'} at 127.30.1.70:37359)
I20240501 19:56:22.383419 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 8f84a882f8e74f2781f337f752516009 from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca at {username='slave'} at 127.30.1.70:37359: session id = 1cd4016c2d8f4feb9fb1fd5a8be29fca-8f84a882f8e74f2781f337f752516009
I20240501 19:56:22.384274 31061 tablet_copy_source_session.cc:205] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:22.385658 1334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a882f8e74f2781f337f752516009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.387459 1334 tablet_copy_client.cc:783] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:22.387615 1334 tablet_copy_client.cc:647] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:22.388499 1336 tablet_copy_client.cc:1002] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:22.415800 1339 ts_tablet_manager.cc:902] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Initiating tablet copy from peer 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:22.416461 1339 tablet_copy_client.cc:308] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:37085
I20240501 19:56:22.420055 31323 tablet_copy_service.cc:140] P 5530341eedde430fbacb4c3ff815159c: Received BeginTabletCopySession request for tablet 174478acaefc4b29a001dd1f900e591b from peer 3393eeb225a4430382811e9bef2948fe ({username='slave'} at 127.30.1.69:51305)
I20240501 19:56:22.420152 31323 tablet_copy_service.cc:161] P 5530341eedde430fbacb4c3ff815159c: Beginning new tablet copy session on tablet 174478acaefc4b29a001dd1f900e591b from peer 3393eeb225a4430382811e9bef2948fe at {username='slave'} at 127.30.1.69:51305: session id = 3393eeb225a4430382811e9bef2948fe-174478acaefc4b29a001dd1f900e591b
I20240501 19:56:22.420778 31323 tablet_copy_source_session.cc:205] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:22.423471 1339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.428231 1339 tablet_copy_client.cc:783] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:22.428403 1339 tablet_copy_client.cc:647] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:22.429455 1341 tablet_copy_client.cc:1002] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:22.657238 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:22.662268 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ac9033765edb40b1ab49d4766eddf1ff: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:23.001207 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:23.055460 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:23.113610 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:23.328083 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:56:23.393862 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 19:56:23.466987 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:23.606282 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:23.822412 1327 tablet_copy_client.cc:523] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:23.823608 1327 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap starting.
I20240501 19:56:23.841091 1327 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:23.841318 1327 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap complete.
I20240501 19:56:23.841454 1327 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.003s
I20240501 19:56:23.841863 1327 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:23.841955 1327 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Initialized, Role: LEARNER
I20240501 19:56:23.842038 1327 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:23.842296 1327 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:23.842696 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session af53e6b6c5e549dda69f656f9a755c5b-33c7d50279364b53953d0a0d45edfb9a received from {username='slave'} at 127.30.1.71:54853
I20240501 19:56:23.842767 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session af53e6b6c5e549dda69f656f9a755c5b-33c7d50279364b53953d0a0d45edfb9a on tablet 33c7d50279364b53953d0a0d45edfb9a with peer af53e6b6c5e549dda69f656f9a755c5b
W20240501 19:56:23.874636 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:56:23.888770 1334 tablet_copy_client.cc:523] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:23.890024 1334 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap starting.
I20240501 19:56:23.907068 1334 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:23.907322 1334 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap complete.
I20240501 19:56:23.907466 1334 ts_tablet_manager.cc:1367] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20240501 19:56:23.907794 1334 raft_consensus.cc:348] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:23.907892 1334 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Initialized, Role: LEARNER
I20240501 19:56:23.907981 1334 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:23.908268 1334 ts_tablet_manager.cc:1398] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240501 19:56:23.908735 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-8f84a882f8e74f2781f337f752516009 received from {username='slave'} at 127.30.1.70:37359
I20240501 19:56:23.908810 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-8f84a882f8e74f2781f337f752516009 on tablet 8f84a882f8e74f2781f337f752516009 with peer 1cd4016c2d8f4feb9fb1fd5a8be29fca
I20240501 19:56:23.929797 1339 tablet_copy_client.cc:523] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:23.931376 1339 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Bootstrap starting.
I20240501 19:56:23.936609 1339 log.cc:824] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:23.954511 1339 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:23.954774 1339 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Bootstrap complete.
I20240501 19:56:23.954998 1339 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 19:56:23.956743 1339 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:23.956964 1339 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3393eeb225a4430382811e9bef2948fe, State: Initialized, Role: LEARNER
I20240501 19:56:23.957237 1339 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:23.957983 1339 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.005s
I20240501 19:56:23.958474 31323 tablet_copy_service.cc:342] P 5530341eedde430fbacb4c3ff815159c: Request end of tablet copy session 3393eeb225a4430382811e9bef2948fe-174478acaefc4b29a001dd1f900e591b received from {username='slave'} at 127.30.1.69:51305
I20240501 19:56:23.958544 31323 tablet_copy_service.cc:433] P 5530341eedde430fbacb4c3ff815159c: ending tablet copy session 3393eeb225a4430382811e9bef2948fe-174478acaefc4b29a001dd1f900e591b on tablet 174478acaefc4b29a001dd1f900e591b with peer 3393eeb225a4430382811e9bef2948fe
W20240501 19:56:23.961701 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb93992159a3461b945d31da3c9bb43f: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:24.261154 32426 raft_consensus.cc:1203] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:24.262251 32540 raft_consensus.cc:1203] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 19:56:24.429965 32295 raft_consensus.cc:1203] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:24.741701 1161 raft_consensus.cc:1050] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca to VOTER
I20240501 19:56:24.742281 1161 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:24.743665 31303 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.743665 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.743870 32426 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.743991 1532 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.744282 1534 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.744434 1501 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.746213 1501 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.746814 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.746840 32426 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.748111 31303 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.749388 30765 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 4 to 5, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:24.752574 31041 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:24.753536 32426 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:24.753635 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:24.753971 1161 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:24.754144 1534 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:24.755656 1532 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.756129 32426 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.756363 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.757833 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f84a882f8e74f2781f337f752516009 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:24.758250 30761 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 5 to 6, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:24.760304 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f84a882f8e74f2781f337f752516009 with delete_type TABLET_DATA_TOMBSTONED (TS 5530341eedde430fbacb4c3ff815159c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46492
I20240501 19:56:24.760821 1542 tablet_replica.cc:330] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: stopping tablet replica
I20240501 19:56:24.761096 1542 raft_consensus.cc:2229] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:24.761292 1542 raft_consensus.cc:2258] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:24.762277 1542 ts_tablet_manager.cc:1875] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:24.763912 1542 ts_tablet_manager.cc:1888] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:24.763984 1542 log.cc:1196] T 8f84a882f8e74f2781f337f752516009 P 5530341eedde430fbacb4c3ff815159c: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f84a882f8e74f2781f337f752516009
I20240501 19:56:24.764477 30755 catalog_manager.cc:4830] TS 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085): tablet 8f84a882f8e74f2781f337f752516009 (table rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]) successfully deleted
I20240501 19:56:24.828006 1161 raft_consensus.cc:1050] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER af53e6b6c5e549dda69f656f9a755c5b to VOTER
I20240501 19:56:24.828348 1161 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:24.829501 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:24.829500 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:24.829536 32426 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:24.829828 1501 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:24.829973 1161 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:24.830169 1534 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:24.831805 1534 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:24.832242 32426 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:24.834548 30759 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 8 to 9, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:24.834590 32540 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:24.834620 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:24.888032 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57626
I20240501 19:56:24.888170 31041 raft_consensus.cc:595] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:24.901716 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57644
I20240501 19:56:24.901808 31041 raft_consensus.cc:595] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:24.960136 1584 raft_consensus.cc:1050] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: attempting to promote NON_VOTER 3393eeb225a4430382811e9bef2948fe to VOTER
I20240501 19:56:24.960575 1584 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:24.961951 31171 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.961977 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.962010 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:24.962349 1586 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.962503 1415 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.962610 1584 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:24.965257 1584 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.965706 31171 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.966444 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.967636 31041 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.968072 30765 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.971148 31303 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:24.972043 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:24.972132 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:24.972434 1415 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:24.972599 1583 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:24.974011 1415 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.974543 31041 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.975675 30755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 174478acaefc4b29a001dd1f900e591b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:24.976116 30759 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 5 to 6, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:24.976760 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:24.978101 31152 tablet_service.cc:1514] Processing DeleteTablet for tablet 174478acaefc4b29a001dd1f900e591b with delete_type TABLET_DATA_TOMBSTONED (TS ac3d66d1d57441db81def86faef0eb4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41114
I20240501 19:56:24.978802 1591 tablet_replica.cc:330] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: stopping tablet replica
I20240501 19:56:24.979209 1591 raft_consensus.cc:2229] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:24.979393 1591 raft_consensus.cc:2258] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:24.980415 1591 ts_tablet_manager.cc:1875] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:24.982091 1591 ts_tablet_manager.cc:1888] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:24.982169 1591 log.cc:1196] T 174478acaefc4b29a001dd1f900e591b P ac3d66d1d57441db81def86faef0eb4a: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/174478acaefc4b29a001dd1f900e591b
I20240501 19:56:24.982642 30753 catalog_manager.cc:4830] TS ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463): tablet 174478acaefc4b29a001dd1f900e591b (table rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]) successfully deleted
W20240501 19:56:25.061550 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 19:56:25.122115 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57660
I20240501 19:56:25.122225 31041 raft_consensus.cc:595] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:25.130169 31171 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:25.131717 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:25.131917 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:25.132313 1614 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:25.132572 1615 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:56:25.132738 31107 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac9033765edb40b1ab49d4766eddf1ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:25.134348 1615 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:25.134927 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:25.137593 30768 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:25.138263 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
W20240501 19:56:25.141685 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:56:25.154974 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57678
I20240501 19:56:25.155063 31041 raft_consensus.cc:595] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:25.213198 1532 raft_consensus.cc:979] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: : Instructing follower 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:25.213336 1532 raft_consensus.cc:1067] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Signalling peer 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:25.213968 32426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a"
dest_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
from {username='slave'} at 127.30.1.66:57655
I20240501 19:56:25.214179 32426 raft_consensus.cc:482] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:25.214323 32426 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.215384 32426 raft_consensus.cc:504] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:25.216435 32426 leader_election.cc:288] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429)
I20240501 19:56:25.217247 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:25.217366 31041 raft_consensus.cc:3032] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:25.217407 31041 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:25.217572 31041 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:25.217793 31041 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.218849 31041 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:25.219907 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:25.220027 31303 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.220826 31303 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:25.221148 32362 leader_election.cc:302] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 1cd4016c2d8f4feb9fb1fd5a8be29fca, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:25.221428 1466 raft_consensus.cc:2781] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:25.221630 1466 raft_consensus.cc:686] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Becoming Leader. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Running, Role: LEADER
I20240501 19:56:25.221894 1466 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:25.223382 32540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c7d50279364b53953d0a0d45edfb9a" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
I20240501 19:56:25.223562 32540 raft_consensus.cc:3037] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.224444 32540 raft_consensus.cc:2454] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:25.224720 30759 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: term changed from 1 to 2, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70). New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:25.389885 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:56:25.568792 1653 ts_tablet_manager.cc:902] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:25.569288 1653 tablet_copy_client.cc:308] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:36463
I20240501 19:56:25.569726 31192 tablet_copy_service.cc:140] P ac3d66d1d57441db81def86faef0eb4a: Received BeginTabletCopySession request for tablet ac9033765edb40b1ab49d4766eddf1ff from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:49045)
I20240501 19:56:25.569859 31192 tablet_copy_service.cc:161] P ac3d66d1d57441db81def86faef0eb4a: Beginning new tablet copy session on tablet ac9033765edb40b1ab49d4766eddf1ff from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:49045: session id = a159fd4137c34800b2f1d142e2272705-ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:25.570771 31192 tablet_copy_source_session.cc:205] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.571957 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.574126 1653 tablet_copy_client.cc:783] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.574287 1653 tablet_copy_client.cc:647] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.575305 1657 tablet_copy_client.cc:1002] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:25.610045 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:25.610471 1466 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 19:56:25.639396 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:56:25.642765 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:56:25.647987 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:56:25.693048 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 19:56:25.693697 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 274800fe27c245139788f075eca1e812: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:25.696123 31041 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:25.696484 1466 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:25.699154 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:25.699967 1636 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 19:56:25.884966 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:56:26.259294 32426 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:26.260356 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:26.260386 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:26.260740 1706 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:26.260926 1707 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:26.262399 1705 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:26.262792 32540 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:26.262984 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:26.264298 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33c7d50279364b53953d0a0d45edfb9a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:26.265012 30759 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:26.267732 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c7d50279364b53953d0a0d45edfb9a with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:26.268523 1714 tablet_replica.cc:330] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:26.268791 1714 raft_consensus.cc:2229] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:26.268961 1714 raft_consensus.cc:2258] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:26.269891 1714 ts_tablet_manager.cc:1875] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:26.271598 1714 ts_tablet_manager.cc:1888] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:56:26.271679 1714 log.cc:1196] T 33c7d50279364b53953d0a0d45edfb9a P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/33c7d50279364b53953d0a0d45edfb9a
I20240501 19:56:26.272168 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet 33c7d50279364b53953d0a0d45edfb9a (table rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]) successfully deleted
I20240501 19:56:26.427819 31041 consensus_queue.cc:237] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:26.429056 31303 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:26.429086 31171 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:26.429492 1733 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:26.430012 1732 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:26.431713 1532 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:26.432087 31303 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:26.434010 31171 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:26.434319 30768 catalog_manager.cc:5484] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index -1 to 4, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
W20240501 19:56:26.437196 30974 consensus_peers.cc:485] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 -> Peer 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745): Couldn't send request to peer 3393eeb225a4430382811e9bef2948fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 940fb959cd9248508c1fa45c7ed84385. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:26.617576 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:56:26.796849 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ac9033765edb40b1ab49d4766eddf1ff: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:26.843672 1760 ts_tablet_manager.cc:902] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:26.844094 1760 tablet_copy_client.cc:308] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:26.847352 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 940fb959cd9248508c1fa45c7ed84385 from peer 3393eeb225a4430382811e9bef2948fe ({username='slave'} at 127.30.1.69:48611)
I20240501 19:56:26.847447 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 940fb959cd9248508c1fa45c7ed84385 from peer 3393eeb225a4430382811e9bef2948fe at {username='slave'} at 127.30.1.69:48611: session id = 3393eeb225a4430382811e9bef2948fe-940fb959cd9248508c1fa45c7ed84385
I20240501 19:56:26.848102 31061 tablet_copy_source_session.cc:205] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:26.848834 1760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940fb959cd9248508c1fa45c7ed84385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:26.850456 1760 tablet_copy_client.cc:783] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:26.850595 1760 tablet_copy_client.cc:647] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:26.851472 1764 tablet_copy_client.cc:1002] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:27.075590 1653 tablet_copy_client.cc:523] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.077178 1653 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:27.097481 1653 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:27.097741 1653 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:27.097887 1653 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 19:56:27.098228 1653 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:27.098325 1653 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:27.098439 1653 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:27.098737 1653 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:27.099213 31192 tablet_copy_service.cc:342] P ac3d66d1d57441db81def86faef0eb4a: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-ac9033765edb40b1ab49d4766eddf1ff received from {username='slave'} at 127.30.1.72:49045
I20240501 19:56:27.099315 31192 tablet_copy_service.cc:433] P ac3d66d1d57441db81def86faef0eb4a: ending tablet copy session a159fd4137c34800b2f1d142e2272705-ac9033765edb40b1ab49d4766eddf1ff on tablet ac9033765edb40b1ab49d4766eddf1ff with peer a159fd4137c34800b2f1d142e2272705
I20240501 19:56:27.339942 32688 raft_consensus.cc:1203] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 19:56:27.721318 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:56:27.793705 1820 raft_consensus.cc:1050] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:27.794088 1820 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
W20240501 19:56:27.795209 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:56:27.795415 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:27.795482 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:27.795483 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:27.795787 1823 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:27.795953 1822 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:27.796100 1758 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:27.797726 1758 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:27.798131 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:27.800207 30759 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:27.800557 32688 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:27.800879 31041 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:27.932886 31171 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44602
I20240501 19:56:27.933074 31171 raft_consensus.cc:595] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.032703 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:56:28.123852 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:56:28.161151 31171 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44622
I20240501 19:56:28.161258 31171 raft_consensus.cc:595] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:28.227205 1758 raft_consensus.cc:979] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: : Instructing follower 17d51481decd4d06964f09c66f981e09 to start an election
I20240501 19:56:28.227308 1758 raft_consensus.cc:1067] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Signalling peer 17d51481decd4d06964f09c66f981e09 to start an election
I20240501 19:56:28.227818 31041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
from {username='slave'} at 127.30.1.67:33405
I20240501 19:56:28.227942 31041 raft_consensus.cc:482] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:28.227990 31041 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.228758 31041 raft_consensus.cc:504] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:28.229202 31041 leader_election.cc:288] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 2 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065), a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:56:28.229516 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:28.229537 32426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
I20240501 19:56:28.229638 31171 raft_consensus.cc:3032] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:28.229681 31171 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:28.229709 32426 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.229844 31171 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:28.230027 31171 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.230525 32426 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
I20240501 19:56:28.230950 31171 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
I20240501 19:56:28.231215 30975 leader_election.cc:302] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 1cd4016c2d8f4feb9fb1fd5a8be29fca, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:28.231379 1732 raft_consensus.cc:2781] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:28.231459 1732 raft_consensus.cc:686] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:28.231593 1732 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:28.233180 32688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a159fd4137c34800b2f1d142e2272705"
I20240501 19:56:28.233345 32688 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.233275 30765 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 1 to 2, leader changed from ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:28.234140 32688 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
W20240501 19:56:28.334868 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:56:28.351752 1760 tablet_copy_client.cc:523] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:28.353093 1760 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Bootstrap starting.
W20240501 19:56:28.353753 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:56:28.373704 1760 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:28.373950 1760 tablet_bootstrap.cc:492] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Bootstrap complete.
I20240501 19:56:28.374105 1760 ts_tablet_manager.cc:1367] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 19:56:28.374590 1760 raft_consensus.cc:348] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:28.374698 1760 raft_consensus.cc:729] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3393eeb225a4430382811e9bef2948fe, State: Initialized, Role: LEARNER
I20240501 19:56:28.374797 1760 consensus_queue.cc:260] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:28.375087 1760 ts_tablet_manager.cc:1398] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:28.375557 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session 3393eeb225a4430382811e9bef2948fe-940fb959cd9248508c1fa45c7ed84385 received from {username='slave'} at 127.30.1.69:48611
I20240501 19:56:28.375633 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session 3393eeb225a4430382811e9bef2948fe-940fb959cd9248508c1fa45c7ed84385 on tablet 940fb959cd9248508c1fa45c7ed84385 with peer 3393eeb225a4430382811e9bef2948fe
W20240501 19:56:28.550422 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:28.780442 31171 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:28.780619 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:28.780817 1732 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:28.781023 1798 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:28.784034 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:28.785434 1887 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 19:56:28.883937 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:56:28.930788 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:56:28.978425 32295 raft_consensus.cc:1203] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:29.290824 31041 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:29.291950 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:29.292047 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:29.292454 1925 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:29.292690 1924 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:29.294236 1887 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:29.294680 32688 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:29.294728 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:29.296528 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac9033765edb40b1ab49d4766eddf1ff with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:29.296872 30765 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:29.299623 31152 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9033765edb40b1ab49d4766eddf1ff with delete_type TABLET_DATA_TOMBSTONED (TS ac3d66d1d57441db81def86faef0eb4a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41114
I20240501 19:56:29.299952 1931 tablet_replica.cc:330] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: stopping tablet replica
I20240501 19:56:29.300097 1931 raft_consensus.cc:2229] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:29.300191 1931 raft_consensus.cc:2258] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:29.300640 1931 ts_tablet_manager.cc:1875] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:29.302461 1931 ts_tablet_manager.cc:1888] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:56:29.302525 1931 log.cc:1196] T ac9033765edb40b1ab49d4766eddf1ff P ac3d66d1d57441db81def86faef0eb4a: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:29.302960 30753 catalog_manager.cc:4830] TS ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463): tablet ac9033765edb40b1ab49d4766eddf1ff (table rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]) successfully deleted
W20240501 19:56:29.386687 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbe3171686254a27a3bbe08786fa7503: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:29.433610 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:29.435209 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:29.435537 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:29.435993 1952 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:29.436208 1586 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:29.437868 1586 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:29.438463 31171 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
W20240501 19:56:29.438623 31236 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429): Couldn't send request to peer af53e6b6c5e549dda69f656f9a755c5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb93992159a3461b945d31da3c9bb43f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:29.440255 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:29.440596 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 8 to 9, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:29.535338 1925 raft_consensus.cc:1050] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER 3393eeb225a4430382811e9bef2948fe to VOTER
I20240501 19:56:29.535691 1925 consensus_queue.cc:237] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:29.536867 31171 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:29.536929 31303 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:29.537148 32295 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:29.537276 1968 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:29.537524 1924 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:29.537642 1887 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:29.539606 1968 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:29.540095 32295 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:29.540359 31171 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:29.541950 31303 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:29.542135 30761 catalog_manager.cc:5484] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 4 to 5, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:29.682063 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57906
I20240501 19:56:29.682178 31041 raft_consensus.cc:595] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:29.897250 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57918
I20240501 19:56:29.897375 31041 raft_consensus.cc:595] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:29.929366 2004 ts_tablet_manager.cc:902] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Initiating tablet copy from peer 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:29.929769 2004 tablet_copy_client.cc:308] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:37085
I20240501 19:56:29.932837 31323 tablet_copy_service.cc:140] P 5530341eedde430fbacb4c3ff815159c: Received BeginTabletCopySession request for tablet bb93992159a3461b945d31da3c9bb43f from peer af53e6b6c5e549dda69f656f9a755c5b ({username='slave'} at 127.30.1.71:56423)
I20240501 19:56:29.932967 31323 tablet_copy_service.cc:161] P 5530341eedde430fbacb4c3ff815159c: Beginning new tablet copy session on tablet bb93992159a3461b945d31da3c9bb43f from peer af53e6b6c5e549dda69f656f9a755c5b at {username='slave'} at 127.30.1.71:56423: session id = af53e6b6c5e549dda69f656f9a755c5b-bb93992159a3461b945d31da3c9bb43f
I20240501 19:56:29.933952 31323 tablet_copy_source_session.cc:205] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:29.934721 2004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:29.936360 2004 tablet_copy_client.cc:783] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:29.936509 2004 tablet_copy_client.cc:647] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:29.937510 2008 tablet_copy_client.cc:1002] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:29.949823 1924 raft_consensus.cc:979] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: : Instructing follower ac3d66d1d57441db81def86faef0eb4a to start an election
I20240501 19:56:29.949918 1924 raft_consensus.cc:1067] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Signalling peer ac3d66d1d57441db81def86faef0eb4a to start an election
I20240501 19:56:29.950311 31171 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
from {username='slave'} at 127.30.1.66:57523
I20240501 19:56:29.950446 31171 raft_consensus.cc:482] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:29.950507 31171 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:29.951315 31171 raft_consensus.cc:504] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:29.951838 31171 leader_election.cc:288] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745)
I20240501 19:56:29.952132 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:29.952172 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:29.952236 31303 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:29.952268 31041 raft_consensus.cc:3032] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:29.952308 31041 raft_consensus.cc:729] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:29.952432 31041 consensus_queue.cc:260] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:29.952584 31041 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:29.953189 31303 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 2.
I20240501 19:56:29.953526 31041 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 2.
I20240501 19:56:29.953786 31107 leader_election.cc:302] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:29.953933 1758 raft_consensus.cc:2781] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:29.954058 1758 raft_consensus.cc:686] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 2 LEADER]: Becoming Leader. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:29.954195 1758 consensus_queue.cc:237] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:29.955893 30761 catalog_manager.cc:5484] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: term changed from 1 to 2, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to ac3d66d1d57441db81def86faef0eb4a (127.30.1.67). New cstate: current_term: 2 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:29.956125 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940fb959cd9248508c1fa45c7ed84385" candidate_uuid: "ac3d66d1d57441db81def86faef0eb4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3393eeb225a4430382811e9bef2948fe"
I20240501 19:56:29.956324 32295 raft_consensus.cc:3037] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:29.957397 32295 raft_consensus.cc:2454] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d66d1d57441db81def86faef0eb4a in term 2.
W20240501 19:56:30.202694 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:56:30.203387 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:56:30.249292 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:56:30.390311 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:56:30.458586 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:56:30.533445 31303 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:30.533840 1758 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:30.536566 32295 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:30.536938 2047 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:30.539707 31041 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:30.540360 1758 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:30.543452 31171 consensus_queue.cc:237] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:30.544643 32295 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:30.544922 31303 raft_consensus.cc:1261] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:30.545017 2047 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:30.545291 1758 consensus_queue.cc:1035] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:30.546566 2051 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:30.546641 32295 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:30.547103 31303 raft_consensus.cc:2932] T 940fb959cd9248508c1fa45c7ed84385 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:30.548489 30753 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 940fb959cd9248508c1fa45c7ed84385 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:30.548681 30761 catalog_manager.cc:5484] T 940fb959cd9248508c1fa45c7ed84385 P ac3d66d1d57441db81def86faef0eb4a reported cstate change: config changed from index 5 to 7, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:30.551244 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet 940fb959cd9248508c1fa45c7ed84385 with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:30.551555 2059 tablet_replica.cc:330] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:30.551719 2059 raft_consensus.cc:2229] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:30.551820 2059 raft_consensus.cc:2258] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:30.552379 2059 ts_tablet_manager.cc:1875] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:30.554090 2059 ts_tablet_manager.cc:1888] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:56:30.554148 2059 log.cc:1196] T 940fb959cd9248508c1fa45c7ed84385 P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/940fb959cd9248508c1fa45c7ed84385
I20240501 19:56:30.554579 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet 940fb959cd9248508c1fa45c7ed84385 (table rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]) successfully deleted
I20240501 19:56:30.747994 31041 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:30.749228 31303 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:30.749527 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 19:56:30.749567 30976 consensus_peers.cc:485] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 274800fe27c245139788f075eca1e812. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:30.749579 2077 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:30.749874 1924 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:30.751221 2077 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:30.751853 31303 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:30.751937 32540 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:30.754083 30765 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 7 to 8, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
W20240501 19:56:30.756458 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:56:31.091611 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:56:31.279356 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:56:31.356359 2114 ts_tablet_manager.cc:902] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:31.356814 2114 tablet_copy_client.cc:308] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:31.360050 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 274800fe27c245139788f075eca1e812 from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:40173)
I20240501 19:56:31.360136 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 274800fe27c245139788f075eca1e812 from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:40173: session id = a159fd4137c34800b2f1d142e2272705-274800fe27c245139788f075eca1e812
I20240501 19:56:31.360796 31061 tablet_copy_source_session.cc:205] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:31.361531 2114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274800fe27c245139788f075eca1e812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:31.363021 2114 tablet_copy_client.cc:783] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:31.363147 2114 tablet_copy_client.cc:647] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:31.364012 2118 tablet_copy_client.cc:1002] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:31.437973 2004 tablet_copy_client.cc:523] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:31.439110 2004 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap starting.
I20240501 19:56:31.455821 2004 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:31.456020 2004 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap complete.
I20240501 19:56:31.456140 2004 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240501 19:56:31.456480 2004 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:31.456579 2004 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Initialized, Role: LEARNER
I20240501 19:56:31.456683 2004 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:31.457046 2004 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:31.457510 31323 tablet_copy_service.cc:342] P 5530341eedde430fbacb4c3ff815159c: Request end of tablet copy session af53e6b6c5e549dda69f656f9a755c5b-bb93992159a3461b945d31da3c9bb43f received from {username='slave'} at 127.30.1.71:56423
I20240501 19:56:31.457587 31323 tablet_copy_service.cc:433] P 5530341eedde430fbacb4c3ff815159c: ending tablet copy session af53e6b6c5e549dda69f656f9a755c5b-bb93992159a3461b945d31da3c9bb43f on tablet bb93992159a3461b945d31da3c9bb43f with peer af53e6b6c5e549dda69f656f9a755c5b
I20240501 19:56:31.775193 32540 raft_consensus.cc:1203] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240501 19:56:32.179150 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 19:56:32.179900 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb93992159a3461b945d31da3c9bb43f: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:32.218432 2171 raft_consensus.cc:1050] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: attempting to promote NON_VOTER af53e6b6c5e549dda69f656f9a755c5b to VOTER
I20240501 19:56:32.218775 2171 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:32.220086 32540 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:32.220084 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:32.220099 31171 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:32.220468 2170 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:32.220631 2036 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:32.220795 2173 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:32.222563 2173 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.223131 31171 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.224206 32540 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.224778 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.225626 30768 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 9 to 10, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:32.229594 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:32.230803 32540 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:32.230894 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:32.231156 2170 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:32.231602 2170 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:32.232652 2036 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.233675 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.234323 30755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb93992159a3461b945d31da3c9bb43f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:32.234889 30759 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 10 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:32.235133 32540 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:32.237390 31152 tablet_service.cc:1514] Processing DeleteTablet for tablet bb93992159a3461b945d31da3c9bb43f with delete_type TABLET_DATA_TOMBSTONED (TS ac3d66d1d57441db81def86faef0eb4a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41114
I20240501 19:56:32.237655 2187 tablet_replica.cc:330] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: stopping tablet replica
I20240501 19:56:32.237876 2187 raft_consensus.cc:2229] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:32.238212 2187 raft_consensus.cc:2258] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:32.238754 2187 ts_tablet_manager.cc:1875] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:32.240641 2187 ts_tablet_manager.cc:1888] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:56:32.240713 2187 log.cc:1196] T bb93992159a3461b945d31da3c9bb43f P ac3d66d1d57441db81def86faef0eb4a: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb93992159a3461b945d31da3c9bb43f
I20240501 19:56:32.241236 30753 catalog_manager.cc:4830] TS ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463): tablet bb93992159a3461b945d31da3c9bb43f (table rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]) successfully deleted
W20240501 19:56:32.690428 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:56:32.749954 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:56:32.853134 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:56:32.864301 2114 tablet_copy_client.cc:523] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:32.865921 2114 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:32.883911 2114 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:32.884142 2114 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:32.884291 2114 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 19:56:32.884667 2114 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:32.884775 2114 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:32.884856 2114 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:32.885236 2114 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:32.885711 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-274800fe27c245139788f075eca1e812 received from {username='slave'} at 127.30.1.72:40173
I20240501 19:56:32.885807 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session a159fd4137c34800b2f1d142e2272705-274800fe27c245139788f075eca1e812 on tablet 274800fe27c245139788f075eca1e812 with peer a159fd4137c34800b2f1d142e2272705
W20240501 19:56:32.961737 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:56:33.186705 32688 raft_consensus.cc:1203] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 19:56:33.326733 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:56:33.554203 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:56:33.554811 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:56:33.759905 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:56:33.809363 2244 raft_consensus.cc:1050] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:33.809710 2244 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:33.810973 32688 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:33.811159 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:33.811174 31301 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:33.811354 2244 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:33.811636 2243 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:33.811792 2246 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:33.813751 2243 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:33.814211 32688 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:33.814426 31301 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:33.816321 32540 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:33.816710 30765 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 8 to 9, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:33.903971 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "274800fe27c245139788f075eca1e812"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57868
I20240501 19:56:33.904090 31041 raft_consensus.cc:595] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:33.926802 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:34.114413 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "274800fe27c245139788f075eca1e812"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57884
I20240501 19:56:34.114670 31041 raft_consensus.cc:595] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.283072 2244 raft_consensus.cc:979] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: : Instructing follower a159fd4137c34800b2f1d142e2272705 to start an election
I20240501 19:56:34.283190 2244 raft_consensus.cc:1067] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Signalling peer a159fd4137c34800b2f1d142e2272705 to start an election
I20240501 19:56:34.283620 32688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "274800fe27c245139788f075eca1e812"
dest_uuid: "a159fd4137c34800b2f1d142e2272705"
from {username='slave'} at 127.30.1.66:49585
I20240501 19:56:34.283795 32688 raft_consensus.cc:482] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:34.283847 32688 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.284619 32688 raft_consensus.cc:504] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:34.285693 32688 leader_election.cc:288] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [CANDIDATE]: Term 2 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429)
I20240501 19:56:34.285879 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "a159fd4137c34800b2f1d142e2272705" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:34.286003 31041 raft_consensus.cc:3032] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:34.286051 31041 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:34.286041 31301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "a159fd4137c34800b2f1d142e2272705" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:34.286159 31301 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.286176 31041 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:34.286336 31041 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.287102 31041 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a159fd4137c34800b2f1d142e2272705 in term 2.
I20240501 19:56:34.287186 31301 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a159fd4137c34800b2f1d142e2272705 in term 2.
I20240501 19:56:34.287523 32624 leader_election.cc:302] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 5530341eedde430fbacb4c3ff815159c, a159fd4137c34800b2f1d142e2272705; no voters:
I20240501 19:56:34.287767 2250 raft_consensus.cc:2781] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:34.287938 2250 raft_consensus.cc:686] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 2 LEADER]: Becoming Leader. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Running, Role: LEADER
I20240501 19:56:34.288200 2250 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:34.290558 32540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274800fe27c245139788f075eca1e812" candidate_uuid: "a159fd4137c34800b2f1d142e2272705" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
I20240501 19:56:34.290659 32540 raft_consensus.cc:3037] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.290746 30768 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 reported cstate change: term changed from 1 to 2, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to a159fd4137c34800b2f1d142e2272705 (127.30.1.72). New cstate: current_term: 2 leader_uuid: "a159fd4137c34800b2f1d142e2272705" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.291479 32540 raft_consensus.cc:2454] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a159fd4137c34800b2f1d142e2272705 in term 2.
I20240501 19:56:34.784421 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer a159fd4137c34800b2f1d142e2272705: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:34.784898 2250 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:34.883226 31301 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer a159fd4137c34800b2f1d142e2272705: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:34.883307 31041 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer a159fd4137c34800b2f1d142e2272705: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:34.883610 2250 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:34.883759 2298 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:34.893306 32688 consensus_queue.cc:237] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:34.894204 31301 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer a159fd4137c34800b2f1d142e2272705: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:34.894228 32540 raft_consensus.cc:1261] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer a159fd4137c34800b2f1d142e2272705: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:34.894621 2298 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:34.894793 2250 consensus_queue.cc:1035] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:34.896230 2304 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:34.896670 31301 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:34.896768 32540 raft_consensus.cc:2932] T 274800fe27c245139788f075eca1e812 P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:34.897965 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 274800fe27c245139788f075eca1e812 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:34.898871 30768 catalog_manager.cc:5484] T 274800fe27c245139788f075eca1e812 P a159fd4137c34800b2f1d142e2272705 reported cstate change: config changed from index 9 to 11, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "a159fd4137c34800b2f1d142e2272705" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.901571 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet 274800fe27c245139788f075eca1e812 with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:34.901858 2310 tablet_replica.cc:330] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:34.901986 2310 raft_consensus.cc:2229] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.902089 2310 raft_consensus.cc:2258] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.902611 2310 ts_tablet_manager.cc:1875] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:34.904415 2310 ts_tablet_manager.cc:1888] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:56:34.904489 2310 log.cc:1196] T 274800fe27c245139788f075eca1e812 P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/274800fe27c245139788f075eca1e812
I20240501 19:56:34.904964 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet 274800fe27c245139788f075eca1e812 (table rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]) successfully deleted
I20240501 19:56:34.990080 31283 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:34.998034 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:34.998404 31021 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:35.000830 31152 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:35.001716 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:35.005746 32406 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:35.007686 32537 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:56:35.037565 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ac9033765edb40b1ab49d4766eddf1ff: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:35.049775 31301 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:35.051192 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:35.051254 31236 consensus_peers.cc:485] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c -> Peer af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429): Couldn't send request to peer af53e6b6c5e549dda69f656f9a755c5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 174478acaefc4b29a001dd1f900e591b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.051545 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:35.051560 2363 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:35.051905 2362 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:35.053588 2170 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:35.054049 31041 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:35.054231 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:35.056474 30762 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 6 to 7, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.063077 31041 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:35.064260 32426 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:35.064260 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:35.064610 2374 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:35.064970 2371 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:35.065168 30974 consensus_peers.cc:485] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 -> Peer 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745): Couldn't send request to peer 3393eeb225a4430382811e9bef2948fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f84a882f8e74f2781f337f752516009. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.066586 2371 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:35.067041 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:35.069162 32426 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:35.069346 30765 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 6 to 7, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:35.076676 31171 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:35.077858 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:35.077862 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 19:56:35.077956 31105 consensus_peers.cc:485] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a -> Peer 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065): Couldn't send request to peer 1cd4016c2d8f4feb9fb1fd5a8be29fca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbe3171686254a27a3bbe08786fa7503. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:35.078224 2384 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:35.078374 2383 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:35.079965 1758 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:35.080435 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:35.082554 30759 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 7 to 8, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
I20240501 19:56:35.086033 32688 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } } }
W20240501 19:56:35.220314 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:56:35.232897 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:56:35.294360 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:56:35.294888 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:56:35.460448 2419 ts_tablet_manager.cc:902] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Initiating tablet copy from peer ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463)
I20240501 19:56:35.460868 2419 tablet_copy_client.cc:308] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:36463
I20240501 19:56:35.461318 31192 tablet_copy_service.cc:140] P ac3d66d1d57441db81def86faef0eb4a: Received BeginTabletCopySession request for tablet dbe3171686254a27a3bbe08786fa7503 from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca ({username='slave'} at 127.30.1.70:50703)
I20240501 19:56:35.461463 31192 tablet_copy_service.cc:161] P ac3d66d1d57441db81def86faef0eb4a: Beginning new tablet copy session on tablet dbe3171686254a27a3bbe08786fa7503 from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca at {username='slave'} at 127.30.1.70:50703: session id = 1cd4016c2d8f4feb9fb1fd5a8be29fca-dbe3171686254a27a3bbe08786fa7503
I20240501 19:56:35.462157 31192 tablet_copy_source_session.cc:205] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.462977 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe3171686254a27a3bbe08786fa7503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.464497 2419 tablet_copy_client.cc:783] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.464617 2419 tablet_copy_client.cc:647] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.465642 2423 tablet_copy_client.cc:1002] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:35.471865 2424 ts_tablet_manager.cc:902] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:35.472322 2424 tablet_copy_client.cc:308] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:35.472790 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 8f84a882f8e74f2781f337f752516009 from peer 3393eeb225a4430382811e9bef2948fe ({username='slave'} at 127.30.1.69:48611)
I20240501 19:56:35.472868 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 8f84a882f8e74f2781f337f752516009 from peer 3393eeb225a4430382811e9bef2948fe at {username='slave'} at 127.30.1.69:48611: session id = 3393eeb225a4430382811e9bef2948fe-8f84a882f8e74f2781f337f752516009
I20240501 19:56:35.473500 31061 tablet_copy_source_session.cc:205] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.474139 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a882f8e74f2781f337f752516009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.475601 2424 tablet_copy_client.cc:783] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.475736 2424 tablet_copy_client.cc:647] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.476678 2428 tablet_copy_client.cc:1002] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:35.548343 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:56:35.647342 2444 ts_tablet_manager.cc:902] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Initiating tablet copy from peer 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:35.647763 2444 tablet_copy_client.cc:308] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:37085
I20240501 19:56:35.648164 31323 tablet_copy_service.cc:140] P 5530341eedde430fbacb4c3ff815159c: Received BeginTabletCopySession request for tablet 174478acaefc4b29a001dd1f900e591b from peer af53e6b6c5e549dda69f656f9a755c5b ({username='slave'} at 127.30.1.71:56423)
I20240501 19:56:35.648250 31323 tablet_copy_service.cc:161] P 5530341eedde430fbacb4c3ff815159c: Beginning new tablet copy session on tablet 174478acaefc4b29a001dd1f900e591b from peer af53e6b6c5e549dda69f656f9a755c5b at {username='slave'} at 127.30.1.71:56423: session id = af53e6b6c5e549dda69f656f9a755c5b-174478acaefc4b29a001dd1f900e591b
I20240501 19:56:35.648905 31323 tablet_copy_source_session.cc:205] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:35.649623 2444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:35.651263 2444 tablet_copy_client.cc:783] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:35.651403 2444 tablet_copy_client.cc:647] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:35.652330 2448 tablet_copy_client.cc:1002] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:35.937245 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:56:36.170354 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:56:36.265875 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:56:36.965929 2419 tablet_copy_client.cc:523] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.967247 2419 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap starting.
I20240501 19:56:36.976951 2424 tablet_copy_client.cc:523] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:36.978302 2424 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Bootstrap starting.
I20240501 19:56:36.989089 2419 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:36.989313 2419 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Bootstrap complete.
I20240501 19:56:36.989451 2419 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.005s
I20240501 19:56:36.989881 2419 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:36.990010 2419 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Initialized, Role: LEARNER
I20240501 19:56:36.990136 2419 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:36.990396 2419 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:36.990909 31192 tablet_copy_service.cc:342] P ac3d66d1d57441db81def86faef0eb4a: Request end of tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-dbe3171686254a27a3bbe08786fa7503 received from {username='slave'} at 127.30.1.70:50703
I20240501 19:56:36.990986 31192 tablet_copy_service.cc:433] P ac3d66d1d57441db81def86faef0eb4a: ending tablet copy session 1cd4016c2d8f4feb9fb1fd5a8be29fca-dbe3171686254a27a3bbe08786fa7503 on tablet dbe3171686254a27a3bbe08786fa7503 with peer 1cd4016c2d8f4feb9fb1fd5a8be29fca
I20240501 19:56:37.000176 2424 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.000430 2424 tablet_bootstrap.cc:492] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Bootstrap complete.
I20240501 19:56:37.000594 2424 ts_tablet_manager.cc:1367] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20240501 19:56:37.000924 2424 raft_consensus.cc:348] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:37.001037 2424 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3393eeb225a4430382811e9bef2948fe, State: Initialized, Role: LEARNER
I20240501 19:56:37.001116 2424 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:37.001375 2424 ts_tablet_manager.cc:1398] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:37.001936 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session 3393eeb225a4430382811e9bef2948fe-8f84a882f8e74f2781f337f752516009 received from {username='slave'} at 127.30.1.69:48611
I20240501 19:56:37.002020 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session 3393eeb225a4430382811e9bef2948fe-8f84a882f8e74f2781f337f752516009 on tablet 8f84a882f8e74f2781f337f752516009 with peer 3393eeb225a4430382811e9bef2948fe
I20240501 19:56:37.050185 32295 raft_consensus.cc:1203] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:37.152578 2444 tablet_copy_client.cc:523] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:37.154055 2444 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap starting.
I20240501 19:56:37.171114 2444 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:37.171350 2444 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap complete.
I20240501 19:56:37.171514 2444 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240501 19:56:37.171881 2444 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:37.171979 2444 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Initialized, Role: LEARNER
I20240501 19:56:37.172078 2444 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:37.172394 2444 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:37.172859 31323 tablet_copy_service.cc:342] P 5530341eedde430fbacb4c3ff815159c: Request end of tablet copy session af53e6b6c5e549dda69f656f9a755c5b-174478acaefc4b29a001dd1f900e591b received from {username='slave'} at 127.30.1.71:56423
I20240501 19:56:37.172963 31323 tablet_copy_service.cc:433] P 5530341eedde430fbacb4c3ff815159c: ending tablet copy session af53e6b6c5e549dda69f656f9a755c5b-174478acaefc4b29a001dd1f900e591b on tablet 174478acaefc4b29a001dd1f900e591b with peer af53e6b6c5e549dda69f656f9a755c5b
I20240501 19:56:37.193889 32540 raft_consensus.cc:1203] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:37.480803 32426 raft_consensus.cc:1203] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 19:56:37.600230 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:56:37.601377 2604 raft_consensus.cc:1050] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER 3393eeb225a4430382811e9bef2948fe to VOTER
I20240501 19:56:37.601735 2604 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:37.603070 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.603098 32295 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.603148 32426 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.603456 2604 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.603688 2607 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.603847 2606 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.605707 2606 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:37.606276 32295 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:37.606334 32426 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:37.608414 30762 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 7 to 8, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.609190 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:37.619665 2496 raft_consensus.cc:1050] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: attempting to promote NON_VOTER af53e6b6c5e549dda69f656f9a755c5b to VOTER
I20240501 19:56:37.619997 2496 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:37.621518 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.621860 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.621877 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 LEARNER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:37.621906 2617 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.622239 2619 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.622890 2617 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:37.623823 2616 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:37.624181 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:37.625604 31041 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:37.625684 32540 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:37.627822 30759 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 7 to 8, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:37.686058 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f84a882f8e74f2781f337f752516009"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58046
I20240501 19:56:37.686157 31041 raft_consensus.cc:595] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:37.694957 31303 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "5530341eedde430fbacb4c3ff815159c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42266
I20240501 19:56:37.695102 31303 raft_consensus.cc:595] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:37.726464 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:56:37.839972 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:56:37.909502 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f84a882f8e74f2781f337f752516009"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58058
I20240501 19:56:37.909615 31041 raft_consensus.cc:595] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:37.918267 31303 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "5530341eedde430fbacb4c3ff815159c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42280
I20240501 19:56:37.918367 31303 raft_consensus.cc:595] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.027415 2607 consensus_queue.cc:579] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.244s)
W20240501 19:56:38.028755 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:56:38.040071 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:56:38.040678 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 274800fe27c245139788f075eca1e812: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:38.060285 2496 raft_consensus.cc:979] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: : Instructing follower 17d51481decd4d06964f09c66f981e09 to start an election
I20240501 19:56:38.060403 2496 raft_consensus.cc:1067] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Signalling peer 17d51481decd4d06964f09c66f981e09 to start an election
I20240501 19:56:38.060837 31041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
from {username='slave'} at 127.30.1.68:54039
I20240501 19:56:38.060974 31041 raft_consensus.cc:482] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.061035 31041 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.061874 31041 raft_consensus.cc:504] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:38.062222 31041 leader_election.cc:288] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 2 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745), af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429)
I20240501 19:56:38.062525 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:38.062582 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3393eeb225a4430382811e9bef2948fe"
I20240501 19:56:38.062646 31303 raft_consensus.cc:3032] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.062629 32540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "17d51481decd4d06964f09c66f981e09" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
I20240501 19:56:38.062685 31303 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Running, Role: LEADER
I20240501 19:56:38.062716 32295 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.062767 32540 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.062839 31303 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:38.063033 31303 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.063658 32295 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
I20240501 19:56:38.063663 32540 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
I20240501 19:56:38.063730 31303 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d51481decd4d06964f09c66f981e09 in term 2.
I20240501 19:56:38.063989 30977 leader_election.cc:302] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 3393eeb225a4430382811e9bef2948fe, 5530341eedde430fbacb4c3ff815159c; no voters:
I20240501 19:56:38.064136 2604 raft_consensus.cc:2781] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.064220 2604 raft_consensus.cc:686] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Becoming Leader. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:38.064388 2604 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:38.065595 2604 consensus_queue.cc:579] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.108s)
I20240501 19:56:38.066155 30761 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 reported cstate change: term changed from 1 to 2, leader changed from 5530341eedde430fbacb4c3ff815159c (127.30.1.68) to 17d51481decd4d06964f09c66f981e09 (127.30.1.66). New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.101469 2678 raft_consensus.cc:1050] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: attempting to promote NON_VOTER 1cd4016c2d8f4feb9fb1fd5a8be29fca to VOTER
I20240501 19:56:38.101909 2678 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:38.103032 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:38.103051 32426 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 LEARNER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:38.103360 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Refusing update from remote peer ac3d66d1d57441db81def86faef0eb4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:38.103379 2677 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.103598 2678 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.103796 2680 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.105727 2680 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:38.106232 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:38.106988 32426 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:38.108347 30761 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 8 to 9, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac3d66d1d57441db81def86faef0eb4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:38.110448 32688 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:38.128383 31171 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:38.128480 31171 raft_consensus.cc:595] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.133733 2604 raft_consensus.cc:979] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: : Instructing follower 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:38.133827 2604 raft_consensus.cc:1067] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Signalling peer 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:38.134202 32426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8f84a882f8e74f2781f337f752516009"
dest_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
from {username='slave'} at 127.30.1.66:57655
I20240501 19:56:38.134318 32426 raft_consensus.cc:482] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.134383 32426 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.135232 32426 raft_consensus.cc:504] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:38.135632 32426 leader_election.cc:288] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Requested vote from peers ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745)
I20240501 19:56:38.135833 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:38.135937 31041 raft_consensus.cc:3032] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.135974 31041 raft_consensus.cc:729] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:38.136049 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:38.136111 31041 consensus_queue.cc:260] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:38.136183 31171 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.136296 31041 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.137071 31171 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.137096 31041 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.137447 32361 leader_election.cc:302] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 1cd4016c2d8f4feb9fb1fd5a8be29fca, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:38.137630 2225 raft_consensus.cc:2781] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.137789 2225 raft_consensus.cc:686] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Becoming Leader. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Running, Role: LEADER
I20240501 19:56:38.137907 2225 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:38.139480 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8f84a882f8e74f2781f337f752516009"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58080
I20240501 19:56:38.139585 31041 raft_consensus.cc:595] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 19:56:38.139652 31041 raft_consensus.cc:603] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 19:56:38.139979 30761 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: term changed from 1 to 2, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70). New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:38.142902 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a882f8e74f2781f337f752516009" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3393eeb225a4430382811e9bef2948fe"
I20240501 19:56:38.143029 32295 raft_consensus.cc:3037] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.143719 32295 raft_consensus.cc:2454] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.147728 2604 consensus_queue.cc:579] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.353s)
I20240501 19:56:38.200011 2678 consensus_queue.cc:579] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.029s)
I20240501 19:56:38.360440 31171 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43176
I20240501 19:56:38.360543 31171 raft_consensus.cc:595] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.447392 31303 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.447798 2606 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.501389 2678 consensus_queue.cc:579] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.403s)
W20240501 19:56:38.574350 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:56:38.575188 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:56:38.579300 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.579322 31171 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503"
dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43190
I20240501 19:56:38.579440 31171 raft_consensus.cc:595] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:38.579651 2606 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.582330 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.582926 2747 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.602917 2678 raft_consensus.cc:979] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: : Instructing follower 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:38.602993 2678 raft_consensus.cc:1067] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Signalling peer 1cd4016c2d8f4feb9fb1fd5a8be29fca to start an election
I20240501 19:56:38.603358 32426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503"
dest_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
from {username='slave'} at 127.30.1.67:55675
I20240501 19:56:38.603466 32426 raft_consensus.cc:482] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:38.603523 32426 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.604240 32426 raft_consensus.cc:504] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:38.604622 32426 leader_election.cc:288] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463), a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:56:38.604838 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:38.604954 31041 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.605012 31171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ac3d66d1d57441db81def86faef0eb4a"
I20240501 19:56:38.605113 31171 raft_consensus.cc:3032] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:38.605159 31171 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac3d66d1d57441db81def86faef0eb4a, State: Running, Role: LEADER
I20240501 19:56:38.605309 31171 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:38.605511 31171 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.605767 31041 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.606431 31171 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.606701 32360 leader_election.cc:302] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 1cd4016c2d8f4feb9fb1fd5a8be29fca, ac3d66d1d57441db81def86faef0eb4a; no voters:
I20240501 19:56:38.606838 2225 raft_consensus.cc:2781] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:38.606927 2225 raft_consensus.cc:686] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Becoming Leader. State: Replica: 1cd4016c2d8f4feb9fb1fd5a8be29fca, State: Running, Role: LEADER
I20240501 19:56:38.607071 2225 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:38.608534 30762 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: term changed from 1 to 2, leader changed from ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) to 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70). New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:38.608857 32688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe3171686254a27a3bbe08786fa7503" candidate_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a159fd4137c34800b2f1d142e2272705"
I20240501 19:56:38.608984 32688 raft_consensus.cc:3037] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:38.609637 32688 raft_consensus.cc:2454] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd4016c2d8f4feb9fb1fd5a8be29fca in term 2.
I20240501 19:56:38.633492 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.633812 2225 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:56:38.643129 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:56:38.658325 32295 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.658648 2225 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.660693 31041 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:38.661037 2766 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:38.662735 32426 consensus_queue.cc:237] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:38.663803 32295 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:38.663846 31171 raft_consensus.cc:1261] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:38.664129 2225 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:38.664275 2766 consensus_queue.cc:1035] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:38.665879 2766 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:38.666473 31171 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:38.666759 32295 raft_consensus.cc:2932] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:38.667392 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f84a882f8e74f2781f337f752516009 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:38.668673 30765 catalog_manager.cc:5484] T 8f84a882f8e74f2781f337f752516009 P 3393eeb225a4430382811e9bef2948fe reported cstate change: config changed from index 8 to 10, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:38.670553 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f84a882f8e74f2781f337f752516009 with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:38.670975 2769 tablet_replica.cc:330] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:38.671134 2769 raft_consensus.cc:2229] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:38.671250 2769 raft_consensus.cc:2258] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:38.671751 2769 ts_tablet_manager.cc:1875] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:38.673450 2769 ts_tablet_manager.cc:1888] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:56:38.673509 2769 log.cc:1196] T 8f84a882f8e74f2781f337f752516009 P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f84a882f8e74f2781f337f752516009
I20240501 19:56:38.673915 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet 8f84a882f8e74f2781f337f752516009 (table rebalance_test_table_2 [id=9146f08f90dd49049851e0a15aa39212]) successfully deleted
W20240501 19:56:38.698972 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:56:38.977370 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:56:39.049170 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:39.049530 2225 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:39.067603 31171 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:39.067991 2225 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:39.070271 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:39.071506 2706 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:39.209215 31041 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:39.210189 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:39.210211 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:39.210489 2807 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:39.210651 2607 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:39.212175 2747 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:39.212556 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:39.212647 32540 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:39.214197 30754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 174478acaefc4b29a001dd1f900e591b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.214756 30761 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 8 to 10, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:39.217099 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet 174478acaefc4b29a001dd1f900e591b with delete_type TABLET_DATA_TOMBSTONED (TS 5530341eedde430fbacb4c3ff815159c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46492
I20240501 19:56:39.217418 2813 tablet_replica.cc:330] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: stopping tablet replica
I20240501 19:56:39.217558 2813 raft_consensus.cc:2229] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.217700 2813 raft_consensus.cc:2258] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.218686 2813 ts_tablet_manager.cc:1875] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.220178 2813 ts_tablet_manager.cc:1888] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:56:39.220237 2813 log.cc:1196] T 174478acaefc4b29a001dd1f900e591b P 5530341eedde430fbacb4c3ff815159c: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/174478acaefc4b29a001dd1f900e591b
I20240501 19:56:39.220633 30755 catalog_manager.cc:4830] TS 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085): tablet 174478acaefc4b29a001dd1f900e591b (table rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]) successfully deleted
I20240501 19:56:39.245389 32426 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:39.246549 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:39.246548 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 19:56:39.246872 32361 consensus_peers.cc:485] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33c7d50279364b53953d0a0d45edfb9a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.246898 2805 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:39.247056 2706 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:39.248625 2766 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:39.248982 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:39.249089 32540 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:39.251124 30762 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 11 to 12, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.257817 31041 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:39.258961 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:39.259011 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 19:56:39.259078 30974 consensus_peers.cc:485] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 -> Peer af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429): Couldn't send request to peer af53e6b6c5e549dda69f656f9a755c5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac9033765edb40b1ab49d4766eddf1ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:39.259343 2747 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:39.259513 2807 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:39.261073 2607 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:39.261531 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:39.263213 32688 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } } }
I20240501 19:56:39.263438 30761 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 11 to 12, NON_VOTER af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) added. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:39.549760 32426 consensus_queue.cc:237] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } }
I20240501 19:56:39.550843 32688 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:39.550873 31041 raft_consensus.cc:1261] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:39.551193 2706 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:39.551391 2805 consensus_queue.cc:1035] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:39.552942 2225 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:39.553318 31041 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:39.554430 32688 raft_consensus.cc:2932] T dbe3171686254a27a3bbe08786fa7503 P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:39.555251 30762 catalog_manager.cc:5484] T dbe3171686254a27a3bbe08786fa7503 P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 9 to 11, VOTER ac3d66d1d57441db81def86faef0eb4a (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } }
I20240501 19:56:39.556493 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbe3171686254a27a3bbe08786fa7503 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:39.557734 31152 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe3171686254a27a3bbe08786fa7503 with delete_type TABLET_DATA_TOMBSTONED (TS ac3d66d1d57441db81def86faef0eb4a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41114
I20240501 19:56:39.558194 2872 tablet_replica.cc:330] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: stopping tablet replica
I20240501 19:56:39.558318 2872 raft_consensus.cc:2229] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:39.558409 2872 raft_consensus.cc:2258] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:39.558952 2872 ts_tablet_manager.cc:1875] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:39.560535 2872 ts_tablet_manager.cc:1888] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:56:39.560590 2872 log.cc:1196] T dbe3171686254a27a3bbe08786fa7503 P ac3d66d1d57441db81def86faef0eb4a: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbe3171686254a27a3bbe08786fa7503
I20240501 19:56:39.561014 30753 catalog_manager.cc:4830] TS ac3d66d1d57441db81def86faef0eb4a (127.30.1.67:36463): tablet dbe3171686254a27a3bbe08786fa7503 (table rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]) successfully deleted
I20240501 19:56:39.637398 2873 ts_tablet_manager.cc:902] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065)
I20240501 19:56:39.637866 2873 tablet_copy_client.cc:308] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:39065
I20240501 19:56:39.641023 32446 tablet_copy_service.cc:140] P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Received BeginTabletCopySession request for tablet 33c7d50279364b53953d0a0d45edfb9a from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:53603)
I20240501 19:56:39.641119 32446 tablet_copy_service.cc:161] P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Beginning new tablet copy session on tablet 33c7d50279364b53953d0a0d45edfb9a from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:53603: session id = a159fd4137c34800b2f1d142e2272705-33c7d50279364b53953d0a0d45edfb9a
I20240501 19:56:39.642096 32446 tablet_copy_source_session.cc:205] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.642967 2873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c7d50279364b53953d0a0d45edfb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.644563 2873 tablet_copy_client.cc:783] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.644707 2873 tablet_copy_client.cc:647] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.645826 2877 tablet_copy_client.cc:1002] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:39.773878 2904 ts_tablet_manager.cc:902] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:39.774298 2904 tablet_copy_client.cc:308] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:39.774693 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet ac9033765edb40b1ab49d4766eddf1ff from peer af53e6b6c5e549dda69f656f9a755c5b ({username='slave'} at 127.30.1.71:54853)
I20240501 19:56:39.774768 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet ac9033765edb40b1ab49d4766eddf1ff from peer af53e6b6c5e549dda69f656f9a755c5b at {username='slave'} at 127.30.1.71:54853: session id = af53e6b6c5e549dda69f656f9a755c5b-ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:39.775528 31061 tablet_copy_source_session.cc:205] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:39.776270 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9033765edb40b1ab49d4766eddf1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:39.777871 2904 tablet_copy_client.cc:783] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:39.778012 2904 tablet_copy_client.cc:647] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:39.778852 2908 tablet_copy_client.cc:1002] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:39.860311 2678 consensus_queue.cc:579] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.232s)
W20240501 19:56:39.861442 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:56:39.914476 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:56:40.022325 2496 consensus_queue.cc:579] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c [LEADER]: Leader has been unable to successfully communicate with peer 00bf2be91e8f4a1a8b6fe07c478bf543 for more than 30 seconds (30.361s)
W20240501 19:56:40.474421 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:56:40.541608 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:56:40.542212 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:56:41.112660 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:56:41.146064 2873 tablet_copy_client.cc:523] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:41.147228 2873 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:41.165107 2873 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:41.165372 2873 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:41.165515 2873 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240501 19:56:41.165885 2873 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:41.165982 2873 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:41.166064 2873 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:41.166288 2873 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:41.166797 32446 tablet_copy_service.cc:342] P 1cd4016c2d8f4feb9fb1fd5a8be29fca: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-33c7d50279364b53953d0a0d45edfb9a received from {username='slave'} at 127.30.1.72:53603
I20240501 19:56:41.166882 32446 tablet_copy_service.cc:433] P 1cd4016c2d8f4feb9fb1fd5a8be29fca: ending tablet copy session a159fd4137c34800b2f1d142e2272705-33c7d50279364b53953d0a0d45edfb9a on tablet 33c7d50279364b53953d0a0d45edfb9a with peer a159fd4137c34800b2f1d142e2272705
W20240501 19:56:41.237771 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:56:41.279287 2904 tablet_copy_client.cc:523] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:41.280808 2904 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap starting.
I20240501 19:56:41.299228 2904 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:41.299495 2904 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Bootstrap complete.
I20240501 19:56:41.299635 2904 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20240501 19:56:41.299968 2904 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:41.300067 2904 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Initialized, Role: LEARNER
I20240501 19:56:41.300168 2904 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: NON_VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: true } }
I20240501 19:56:41.300460 2904 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 19:56:41.300873 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session af53e6b6c5e549dda69f656f9a755c5b-ac9033765edb40b1ab49d4766eddf1ff received from {username='slave'} at 127.30.1.71:54853
I20240501 19:56:41.300964 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session af53e6b6c5e549dda69f656f9a755c5b-ac9033765edb40b1ab49d4766eddf1ff on tablet ac9033765edb40b1ab49d4766eddf1ff with peer af53e6b6c5e549dda69f656f9a755c5b
W20240501 19:56:41.419718 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:56:41.520982 32688 raft_consensus.cc:1203] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 19:56:41.557674 32540 raft_consensus.cc:1203] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 19:56:41.924894 3029 raft_consensus.cc:1050] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:41.925377 3029 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:41.926873 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:41.926873 31303 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:41.927193 3031 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:41.927235 32688 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:41.927557 3028 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:41.927733 2926 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:41.929443 3028 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.929891 32688 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.930384 31303 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.931797 32540 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.932221 30765 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 reported cstate change: config changed from index 12 to 13, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.935866 32426 consensus_queue.cc:237] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:41.936794 32540 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 19:56:41.936892 32688 raft_consensus.cc:1261] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 1cd4016c2d8f4feb9fb1fd5a8be29fca: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 19:56:41.937170 2926 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:41.937319 3029 consensus_queue.cc:1035] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:41.938637 3029 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.939069 32540 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.939190 32688 raft_consensus.cc:2932] T 33c7d50279364b53953d0a0d45edfb9a P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:41.940281 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33c7d50279364b53953d0a0d45edfb9a with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:41.940886 30761 catalog_manager.cc:5484] T 33c7d50279364b53953d0a0d45edfb9a P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 13 to 14, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:41.942837 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c7d50279364b53953d0a0d45edfb9a with delete_type TABLET_DATA_TOMBSTONED (TS 5530341eedde430fbacb4c3ff815159c not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:46492
I20240501 19:56:41.943132 3039 tablet_replica.cc:330] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: stopping tablet replica
I20240501 19:56:41.943256 3039 raft_consensus.cc:2229] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:41.943445 3039 raft_consensus.cc:2258] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:41.943986 3039 ts_tablet_manager.cc:1875] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:41.945501 3039 ts_tablet_manager.cc:1888] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240501 19:56:41.945575 3039 log.cc:1196] T 33c7d50279364b53953d0a0d45edfb9a P 5530341eedde430fbacb4c3ff815159c: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/33c7d50279364b53953d0a0d45edfb9a
I20240501 19:56:41.945992 30755 catalog_manager.cc:4830] TS 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085): tablet 33c7d50279364b53953d0a0d45edfb9a (table rebalance_test_table_0 [id=8169c68d62ae45b89f16163fce33682d]) successfully deleted
I20240501 19:56:42.045687 31303 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:42.046891 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:42.046979 32540 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:42.047250 3059 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:56:42.047438 31236 consensus_peers.cc:485] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c -> Peer 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745): Couldn't send request to peer 3393eeb225a4430382811e9bef2948fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb93992159a3461b945d31da3c9bb43f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:42.047442 2921 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:42.049052 2921 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:42.049613 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:42.049696 32540 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } } }
I20240501 19:56:42.051187 30762 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 11 to 12, NON_VOTER 3393eeb225a4430382811e9bef2948fe (127.30.1.69) added. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:42.112016 3072 raft_consensus.cc:1050] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER af53e6b6c5e549dda69f656f9a755c5b to VOTER
I20240501 19:56:42.112484 3072 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:42.113799 32540 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:42.113852 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:42.113868 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:42.114215 3075 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:42.114400 3072 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:42.114533 3074 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:42.116148 3075 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:42.116578 32540 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:42.116617 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:42.118507 30761 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 12 to 13, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:42.118567 32688 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:42.276252 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33670
I20240501 19:56:42.276368 31041 raft_consensus.cc:595] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Received request to transfer leadership
W20240501 19:56:42.316102 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:56:42.443271 3095 ts_tablet_manager.cc:902] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Initiating tablet copy from peer 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085)
I20240501 19:56:42.443768 3095 tablet_copy_client.cc:308] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:37085
I20240501 19:56:42.444259 31323 tablet_copy_service.cc:140] P 5530341eedde430fbacb4c3ff815159c: Received BeginTabletCopySession request for tablet bb93992159a3461b945d31da3c9bb43f from peer 3393eeb225a4430382811e9bef2948fe ({username='slave'} at 127.30.1.69:51305)
I20240501 19:56:42.444355 31323 tablet_copy_service.cc:161] P 5530341eedde430fbacb4c3ff815159c: Beginning new tablet copy session on tablet bb93992159a3461b945d31da3c9bb43f from peer 3393eeb225a4430382811e9bef2948fe at {username='slave'} at 127.30.1.69:51305: session id = 3393eeb225a4430382811e9bef2948fe-bb93992159a3461b945d31da3c9bb43f
I20240501 19:56:42.445255 31323 tablet_copy_source_session.cc:205] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:42.446012 3095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb93992159a3461b945d31da3c9bb43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.447651 3095 tablet_copy_client.cc:783] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:42.447789 3095 tablet_copy_client.cc:647] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:42.448758 3099 tablet_copy_client.cc:1002] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: tablet copy: Injecting latency into file download: 1500
I20240501 19:56:42.489866 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33684
I20240501 19:56:42.489972 31041 raft_consensus.cc:595] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Received request to transfer leadership
W20240501 19:56:42.581763 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:56:42.616003 3072 raft_consensus.cc:979] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: : Instructing follower af53e6b6c5e549dda69f656f9a755c5b to start an election
I20240501 19:56:42.616094 3072 raft_consensus.cc:1067] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Signalling peer af53e6b6c5e549dda69f656f9a755c5b to start an election
I20240501 19:56:42.616520 32540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff"
dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
from {username='slave'} at 127.30.1.66:57343
I20240501 19:56:42.616709 32540 raft_consensus.cc:482] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:42.616773 32540 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:42.617586 32540 raft_consensus.cc:504] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:42.618618 32540 leader_election.cc:288] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [CANDIDATE]: Term 3 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), 1cd4016c2d8f4feb9fb1fd5a8be29fca (127.30.1.70:39065), a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:56:42.618939 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:42.619045 31041 raft_consensus.cc:3032] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:56:42.619083 31041 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:42.619241 31041 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:42.619455 31041 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:42.620354 31041 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
I20240501 19:56:42.622494 32426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca"
I20240501 19:56:42.622597 32426 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:42.622597 32688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9033765edb40b1ab49d4766eddf1ff" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "a159fd4137c34800b2f1d142e2272705"
I20240501 19:56:42.622716 32688 raft_consensus.cc:3037] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:42.623288 32426 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
I20240501 19:56:42.623564 32490 leader_election.cc:302] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 1cd4016c2d8f4feb9fb1fd5a8be29fca, af53e6b6c5e549dda69f656f9a755c5b; no voters:
I20240501 19:56:42.623615 32688 raft_consensus.cc:2454] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
I20240501 19:56:42.623917 3121 raft_consensus.cc:2781] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:42.624074 3121 raft_consensus.cc:686] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 3 LEADER]: Becoming Leader. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Running, Role: LEADER
I20240501 19:56:42.624341 3121 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:42.626184 30762 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b reported cstate change: term changed from 2 to 3, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71). New cstate: current_term: 3 leader_uuid: "af53e6b6c5e549dda69f656f9a755c5b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:42.975668 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:56:43.015853 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 33c7d50279364b53953d0a0d45edfb9a: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:43.024960 31041 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:43.025378 3121 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:43.027784 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:43.028131 3121 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:43.030587 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:43.031033 3151 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 19:56:43.161387 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:56:43.556830 32540 consensus_queue.cc:237] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } }
I20240501 19:56:43.557907 32426 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 19:56:43.557946 32688 raft_consensus.cc:1261] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 19:56:43.558348 3190 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:43.558555 3191 consensus_queue.cc:1035] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:43.560082 3190 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P af53e6b6c5e549dda69f656f9a755c5b [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:43.560519 32688 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:43.560842 32426 raft_consensus.cc:2932] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:43.562965 30765 catalog_manager.cc:5484] T ac9033765edb40b1ab49d4766eddf1ff P 1cd4016c2d8f4feb9fb1fd5a8be29fca reported cstate change: config changed from index 13 to 15, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 3 leader_uuid: "af53e6b6c5e549dda69f656f9a755c5b" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } }
I20240501 19:56:43.565387 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac9033765edb40b1ab49d4766eddf1ff with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:43.565800 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9033765edb40b1ab49d4766eddf1ff with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:43.566082 3196 tablet_replica.cc:330] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:43.566210 3196 raft_consensus.cc:2229] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:43.566329 3196 raft_consensus.cc:2258] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:43.566965 3196 ts_tablet_manager.cc:1875] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:43.568624 3196 ts_tablet_manager.cc:1888] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240501 19:56:43.568691 3196 log.cc:1196] T ac9033765edb40b1ab49d4766eddf1ff P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:43.569164 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet ac9033765edb40b1ab49d4766eddf1ff (table rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]) successfully deleted
W20240501 19:56:43.684120 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:56:43.696163 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 19:56:43.696718 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:56:43.914115 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:56:43.949158 3095 tablet_copy_client.cc:523] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:43.950448 3095 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Bootstrap starting.
I20240501 19:56:43.970656 3095 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:43.970933 3095 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Bootstrap complete.
I20240501 19:56:43.971082 3095 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 19:56:43.971415 3095 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:43.971536 3095 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3393eeb225a4430382811e9bef2948fe, State: Initialized, Role: LEARNER
I20240501 19:56:43.971619 3095 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: NON_VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: true } }
I20240501 19:56:43.971873 3095 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:43.972298 31323 tablet_copy_service.cc:342] P 5530341eedde430fbacb4c3ff815159c: Request end of tablet copy session 3393eeb225a4430382811e9bef2948fe-bb93992159a3461b945d31da3c9bb43f received from {username='slave'} at 127.30.1.69:51305
I20240501 19:56:43.972365 31323 tablet_copy_service.cc:433] P 5530341eedde430fbacb4c3ff815159c: ending tablet copy session 3393eeb225a4430382811e9bef2948fe-bb93992159a3461b945d31da3c9bb43f on tablet bb93992159a3461b945d31da3c9bb43f with peer 3393eeb225a4430382811e9bef2948fe
W20240501 19:56:44.100225 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:44.587461 32295 raft_consensus.cc:1203] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240501 19:56:44.656210 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:56:45.108218 3251 raft_consensus.cc:1050] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: attempting to promote NON_VOTER 3393eeb225a4430382811e9bef2948fe to VOTER
I20240501 19:56:45.108575 3251 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:45.109812 32295 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 LEARNER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:45.109849 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:45.109867 32540 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 5530341eedde430fbacb4c3ff815159c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:45.110230 3253 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:45.110404 3251 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:45.110539 3177 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:45.112643 3253 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:45.113116 32295 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:45.114533 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:45.114749 32540 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:45.115056 30765 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c reported cstate change: config changed from index 12 to 13, 3393eeb225a4430382811e9bef2948fe (127.30.1.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5530341eedde430fbacb4c3ff815159c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:45.205631 31303 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f"
dest_uuid: "5530341eedde430fbacb4c3ff815159c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53954
I20240501 19:56:45.205741 31303 raft_consensus.cc:595] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Received request to transfer leadership
W20240501 19:56:45.211146 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:56:45.415393 31303 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f"
dest_uuid: "5530341eedde430fbacb4c3ff815159c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53974
I20240501 19:56:45.415503 31303 raft_consensus.cc:595] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:45.525195 3251 raft_consensus.cc:979] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: : Instructing follower af53e6b6c5e549dda69f656f9a755c5b to start an election
I20240501 19:56:45.525339 3251 raft_consensus.cc:1067] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Signalling peer af53e6b6c5e549dda69f656f9a755c5b to start an election
I20240501 19:56:45.525703 32540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f"
dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
from {username='slave'} at 127.30.1.68:58671
I20240501 19:56:45.525820 32540 raft_consensus.cc:482] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:45.525884 32540 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:45.526649 32540 raft_consensus.cc:504] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:45.527101 32540 leader_election.cc:288] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [CANDIDATE]: Term 3 election: Requested vote from peers 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085), a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409), 3393eeb225a4430382811e9bef2948fe (127.30.1.69:45745)
I20240501 19:56:45.527338 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "5530341eedde430fbacb4c3ff815159c"
I20240501 19:56:45.527405 32688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "a159fd4137c34800b2f1d142e2272705"
I20240501 19:56:45.527457 31303 raft_consensus.cc:3032] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:56:45.527503 31303 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5530341eedde430fbacb4c3ff815159c, State: Running, Role: LEADER
I20240501 19:56:45.527519 32688 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:45.527666 31303 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:45.527875 31303 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:45.528391 32688 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
I20240501 19:56:45.528935 31303 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
I20240501 19:56:45.529223 32493 leader_election.cc:302] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5530341eedde430fbacb4c3ff815159c, a159fd4137c34800b2f1d142e2272705, af53e6b6c5e549dda69f656f9a755c5b; no voters:
I20240501 19:56:45.529384 3121 raft_consensus.cc:2781] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:45.529539 3121 raft_consensus.cc:686] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 3 LEADER]: Becoming Leader. State: Replica: af53e6b6c5e549dda69f656f9a755c5b, State: Running, Role: LEADER
I20240501 19:56:45.529678 3121 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:45.531229 32295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb93992159a3461b945d31da3c9bb43f" candidate_uuid: "af53e6b6c5e549dda69f656f9a755c5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "3393eeb225a4430382811e9bef2948fe"
I20240501 19:56:45.531364 32295 raft_consensus.cc:3037] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:45.531276 30761 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b reported cstate change: term changed from 2 to 3, leader changed from 5530341eedde430fbacb4c3ff815159c (127.30.1.68) to af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71). New cstate: current_term: 3 leader_uuid: "af53e6b6c5e549dda69f656f9a755c5b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.532207 32295 raft_consensus.cc:2454] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af53e6b6c5e549dda69f656f9a755c5b in term 3.
W20240501 19:56:45.652446 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:56:45.653050 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:56:45.707911 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:56:45.797777 30753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:56:45.798449 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbe3171686254a27a3bbe08786fa7503: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
I20240501 19:56:45.983796 31303 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:45.984197 3121 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:46.014937 32295 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:46.015342 3121 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:46.017844 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:46.018347 3121 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 19:56:46.131942 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:56:46.219413 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:56:46.500281 32540 consensus_queue.cc:237] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } }
I20240501 19:56:46.501377 32688 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 19:56:46.501377 32295 raft_consensus.cc:1261] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Refusing update from remote peer af53e6b6c5e549dda69f656f9a755c5b: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240501 19:56:46.501704 3302 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:46.501899 3121 consensus_queue.cc:1035] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:46.503211 3121 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:46.503569 32295 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:46.505010 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb93992159a3461b945d31da3c9bb43f with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:46.505143 32688 raft_consensus.cc:2932] T bb93992159a3461b945d31da3c9bb43f P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } }
I20240501 19:56:46.505761 30759 catalog_manager.cc:5484] T bb93992159a3461b945d31da3c9bb43f P af53e6b6c5e549dda69f656f9a755c5b reported cstate change: config changed from index 13 to 15, VOTER 5530341eedde430fbacb4c3ff815159c (127.30.1.68) evicted. New cstate: current_term: 3 leader_uuid: "af53e6b6c5e549dda69f656f9a755c5b" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.508338 31283 tablet_service.cc:1514] Processing DeleteTablet for tablet bb93992159a3461b945d31da3c9bb43f with delete_type TABLET_DATA_TOMBSTONED (TS 5530341eedde430fbacb4c3ff815159c not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:46492
I20240501 19:56:46.508636 3326 tablet_replica.cc:330] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: stopping tablet replica
I20240501 19:56:46.508746 3326 raft_consensus.cc:2229] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:46.508842 3326 raft_consensus.cc:2258] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:46.509385 3326 ts_tablet_manager.cc:1875] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:46.511071 3326 ts_tablet_manager.cc:1888] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240501 19:56:46.511135 3326 log.cc:1196] T bb93992159a3461b945d31da3c9bb43f P 5530341eedde430fbacb4c3ff815159c: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb93992159a3461b945d31da3c9bb43f
I20240501 19:56:46.511555 30755 catalog_manager.cc:4830] TS 5530341eedde430fbacb4c3ff815159c (127.30.1.68:37085): tablet bb93992159a3461b945d31da3c9bb43f (table rebalance_test_table_1 [id=6f2d9c7532d241fe920fdeac80eb5724]) successfully deleted
W20240501 19:56:46.618378 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:56:46.700758 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:46.700942 31021 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:46.711272 32406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:46.714260 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:46.727833 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:46.728528 32537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:46.729413 32275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:46.757417 31041 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:46.759105 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:46.759110 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:46.759677 3377 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 19:56:46.759845 30976 consensus_peers.cc:485] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 -> Peer a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409): Couldn't send request to peer a159fd4137c34800b2f1d142e2272705. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 174478acaefc4b29a001dd1f900e591b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:46.759868 3378 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:46.761389 3377 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:46.761790 32540 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:46.761956 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
I20240501 19:56:46.764129 30759 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe reported cstate change: config changed from index 10 to 11, NON_VOTER a159fd4137c34800b2f1d142e2272705 (127.30.1.72) added. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } } }
W20240501 19:56:47.101070 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:56:47.335400 3397 ts_tablet_manager.cc:902] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Initiating tablet copy from peer 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547)
I20240501 19:56:47.335853 3397 tablet_copy_client.cc:308] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:37547
I20240501 19:56:47.336294 31061 tablet_copy_service.cc:140] P 17d51481decd4d06964f09c66f981e09: Received BeginTabletCopySession request for tablet 174478acaefc4b29a001dd1f900e591b from peer a159fd4137c34800b2f1d142e2272705 ({username='slave'} at 127.30.1.72:40173)
I20240501 19:56:47.336386 31061 tablet_copy_service.cc:161] P 17d51481decd4d06964f09c66f981e09: Beginning new tablet copy session on tablet 174478acaefc4b29a001dd1f900e591b from peer a159fd4137c34800b2f1d142e2272705 at {username='slave'} at 127.30.1.72:40173: session id = a159fd4137c34800b2f1d142e2272705-174478acaefc4b29a001dd1f900e591b
I20240501 19:56:47.337093 31061 tablet_copy_source_session.cc:205] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:47.337869 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174478acaefc4b29a001dd1f900e591b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:47.339398 3397 tablet_copy_client.cc:783] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:47.339521 3397 tablet_copy_client.cc:647] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:47.340387 3401 tablet_copy_client.cc:1002] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: tablet copy: Injecting latency into file download: 1500
W20240501 19:56:47.771749 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:56:48.066416 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:56:48.439576 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:56:48.582152 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:56:48.595816 30753 catalog_manager.cc:4573] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb93992159a3461b945d31da3c9bb43f: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111)
W20240501 19:56:48.781113 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:56:48.840664 3397 tablet_copy_client.cc:523] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:48.841943 3397 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Bootstrap starting.
I20240501 19:56:48.860280 3397 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:48.860503 3397 tablet_bootstrap.cc:492] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Bootstrap complete.
I20240501 19:56:48.860623 3397 ts_tablet_manager.cc:1367] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240501 19:56:48.860950 3397 raft_consensus.cc:348] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:48.861052 3397 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a159fd4137c34800b2f1d142e2272705, State: Initialized, Role: LEARNER
I20240501 19:56:48.861129 3397 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: NON_VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: true } }
I20240501 19:56:48.861379 3397 ts_tablet_manager.cc:1398] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:48.861806 31061 tablet_copy_service.cc:342] P 17d51481decd4d06964f09c66f981e09: Request end of tablet copy session a159fd4137c34800b2f1d142e2272705-174478acaefc4b29a001dd1f900e591b received from {username='slave'} at 127.30.1.72:40173
I20240501 19:56:48.861886 31061 tablet_copy_service.cc:433] P 17d51481decd4d06964f09c66f981e09: ending tablet copy session a159fd4137c34800b2f1d142e2272705-174478acaefc4b29a001dd1f900e591b on tablet 174478acaefc4b29a001dd1f900e591b with peer a159fd4137c34800b2f1d142e2272705
W20240501 19:56:48.886436 30975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:48.924077 32688 raft_consensus.cc:1203] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240501 19:56:48.989181 30975 consensus_peers.cc:485] T 6c061670c33145f389f4767a355537cf P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:56:49.166376 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:56:49.350075 3437 raft_consensus.cc:1050] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: attempting to promote NON_VOTER a159fd4137c34800b2f1d142e2272705 to VOTER
I20240501 19:56:49.350451 3437 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:49.351686 32688 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 LEARNER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:49.351752 32295 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:49.351774 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Refusing update from remote peer 17d51481decd4d06964f09c66f981e09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:49.352069 3464 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:49.352267 3466 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:49.352458 3467 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:49.354216 3466 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:49.354660 32295 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:49.354768 32540 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:49.355913 32688 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:49.357069 30768 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 reported cstate change: config changed from index 11 to 12, a159fd4137c34800b2f1d142e2272705 (127.30.1.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "17d51481decd4d06964f09c66f981e09" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 19:56:49.397812 31106 consensus_peers.cc:485] T 2edc89ebb6984377ab459e5e219a4fab P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:56:49.451747 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33838
I20240501 19:56:49.451855 31041 raft_consensus.cc:595] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:49.661975 31041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "17d51481decd4d06964f09c66f981e09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33862
I20240501 19:56:49.662102 31041 raft_consensus.cc:595] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:49.758920 3464 raft_consensus.cc:979] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: : Instructing follower 3393eeb225a4430382811e9bef2948fe to start an election
I20240501 19:56:49.759022 3464 raft_consensus.cc:1067] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Signalling peer 3393eeb225a4430382811e9bef2948fe to start an election
I20240501 19:56:49.759469 32295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b"
dest_uuid: "3393eeb225a4430382811e9bef2948fe"
from {username='slave'} at 127.30.1.66:58077
I20240501 19:56:49.759634 32295 raft_consensus.cc:482] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:49.759707 32295 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:49.760449 32295 raft_consensus.cc:504] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:49.761471 32295 leader_election.cc:288] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [CANDIDATE]: Term 3 election: Requested vote from peers 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547), af53e6b6c5e549dda69f656f9a755c5b (127.30.1.71:37429), a159fd4137c34800b2f1d142e2272705 (127.30.1.72:45409)
I20240501 19:56:49.761690 31041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "3393eeb225a4430382811e9bef2948fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "17d51481decd4d06964f09c66f981e09"
I20240501 19:56:49.761798 31041 raft_consensus.cc:3032] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:56:49.761839 31041 raft_consensus.cc:729] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 17d51481decd4d06964f09c66f981e09, State: Running, Role: LEADER
I20240501 19:56:49.761960 31041 consensus_queue.cc:260] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:49.762143 31041 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:49.762799 31041 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3393eeb225a4430382811e9bef2948fe in term 3.
I20240501 19:56:49.764667 32540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "3393eeb225a4430382811e9bef2948fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "af53e6b6c5e549dda69f656f9a755c5b"
I20240501 19:56:49.764767 32540 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:49.765511 32540 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3393eeb225a4430382811e9bef2948fe in term 3.
I20240501 19:56:49.765790 32228 leader_election.cc:302] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 17d51481decd4d06964f09c66f981e09, 3393eeb225a4430382811e9bef2948fe, af53e6b6c5e549dda69f656f9a755c5b; no voters:
I20240501 19:56:49.765983 3471 raft_consensus.cc:2781] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:49.765983 32688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174478acaefc4b29a001dd1f900e591b" candidate_uuid: "3393eeb225a4430382811e9bef2948fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "a159fd4137c34800b2f1d142e2272705"
I20240501 19:56:49.766088 32688 raft_consensus.cc:3037] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:49.766157 3471 raft_consensus.cc:686] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 3 LEADER]: Becoming Leader. State: Replica: 3393eeb225a4430382811e9bef2948fe, State: Running, Role: LEADER
I20240501 19:56:49.766409 3471 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:49.766805 32688 raft_consensus.cc:2454] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3393eeb225a4430382811e9bef2948fe in term 3.
I20240501 19:56:49.768323 30765 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe reported cstate change: term changed from 2 to 3, leader changed from 17d51481decd4d06964f09c66f981e09 (127.30.1.66) to 3393eeb225a4430382811e9bef2948fe (127.30.1.69). New cstate: current_term: 3 leader_uuid: "3393eeb225a4430382811e9bef2948fe" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.151266 31041 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Refusing update from remote peer 3393eeb225a4430382811e9bef2948fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 19:56:50.151703 3471 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:50.261992 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Refusing update from remote peer 3393eeb225a4430382811e9bef2948fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 19:56:50.262388 3471 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:50.265575 32688 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer 3393eeb225a4430382811e9bef2948fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 19:56:50.265964 3515 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 19:56:50.290647 31106 consensus_peers.cc:485] T 0684929c231d45a086b6356a9e6922ec P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:56:50.589892 31237 consensus_peers.cc:485] T b2ccd7f3fea4473497944de737911828 P 5530341eedde430fbacb4c3ff815159c -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:56:50.773785 30975 consensus_peers.cc:485] T 9de501bd2ff545f9842d386a2ea8fba8 P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:56:50.833832 32295 consensus_queue.cc:237] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } }
I20240501 19:56:50.834893 32688 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Refusing update from remote peer 3393eeb225a4430382811e9bef2948fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:50.834909 32540 raft_consensus.cc:1261] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Refusing update from remote peer 3393eeb225a4430382811e9bef2948fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:50.835314 3536 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:50.835547 3535 consensus_queue.cc:1035] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:50.836998 3515 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P 3393eeb225a4430382811e9bef2948fe [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:50.837499 32540 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P af53e6b6c5e549dda69f656f9a755c5b [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:50.837499 32688 raft_consensus.cc:2932] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:50.839088 30752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 174478acaefc4b29a001dd1f900e591b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:50.839906 30759 catalog_manager.cc:5484] T 174478acaefc4b29a001dd1f900e591b P a159fd4137c34800b2f1d142e2272705 reported cstate change: config changed from index 12 to 14, VOTER 17d51481decd4d06964f09c66f981e09 (127.30.1.66) evicted. New cstate: current_term: 3 leader_uuid: "3393eeb225a4430382811e9bef2948fe" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "3393eeb225a4430382811e9bef2948fe" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 45745 } attrs { promote: false } } peers { permanent_uuid: "af53e6b6c5e549dda69f656f9a755c5b" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 37429 } attrs { promote: false } } peers { permanent_uuid: "a159fd4137c34800b2f1d142e2272705" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 45409 } attrs { promote: false } } }
I20240501 19:56:50.842569 31021 tablet_service.cc:1514] Processing DeleteTablet for tablet 174478acaefc4b29a001dd1f900e591b with delete_type TABLET_DATA_TOMBSTONED (TS 17d51481decd4d06964f09c66f981e09 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:53434
I20240501 19:56:50.842866 3542 tablet_replica.cc:330] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: stopping tablet replica
I20240501 19:56:50.842991 3542 raft_consensus.cc:2229] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.843125 3542 raft_consensus.cc:2258] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.843767 3542 ts_tablet_manager.cc:1875] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:50.845522 3542 ts_tablet_manager.cc:1888] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240501 19:56:50.845580 3542 log.cc:1196] T 174478acaefc4b29a001dd1f900e591b P 17d51481decd4d06964f09c66f981e09: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/174478acaefc4b29a001dd1f900e591b
I20240501 19:56:50.846002 30754 catalog_manager.cc:4830] TS 17d51481decd4d06964f09c66f981e09 (127.30.1.66:37547): tablet 174478acaefc4b29a001dd1f900e591b (table rebalance_test_table_3 [id=ddd9bed2bb8240c38261ba1de9537ecc]) successfully deleted
I20240501 19:56:50.949045 32406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.949252 32275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:50.952412 31021 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:50.953356 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:50.957231 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:50.962211 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:50.980088 32537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:51.051204 32406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.051452 31021 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.055486 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:51.057132 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.059952 32537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:51.062980 32275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.078680 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.113559 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.65:43707
--local_ip_for_outbound_sockets=127.30.1.65
--tserver_master_addrs=127.30.1.126:46341
--webserver_port=33291
--webserver_interface=127.30.1.65
--builtin_ntp_servers=127.30.1.84:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 19:56:51.126916 31237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:56:51.137171 30975 consensus_peers.cc:485] T 4f614c0ec5274c31a99bff11fe40307f P 17d51481decd4d06964f09c66f981e09 -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:56:51.173656 31106 consensus_peers.cc:485] T 33ae5913b9cc43c1a82b2d03646983c5 P ac3d66d1d57441db81def86faef0eb4a -> Peer 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): Couldn't send request to peer 00bf2be91e8f4a1a8b6fe07c478bf543. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.65:43707: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:56:51.215281 3621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:51.215471 3621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:51.215529 3621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:51.218503 3621 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:51.218569 3621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:51.218667 3621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.65
I20240501 19:56:51.222131 3621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.65:43707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.1.65
--webserver_port=33291
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.30.1.126:46341
--never_fsync=true
--heap_profile_path=/tmp/kudu.3621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.65
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:51.222826 3621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:51.223884 3621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:51.231945 3629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:51.232656 3621 server_base.cc:1034] running on GCE node
W20240501 19:56:51.233289 3628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:51.233883 3631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:51.234189 3621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:51.234642 3621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:51.235780 3621 hybrid_clock.cc:648] HybridClock initialized: now 1714593411235769 us; error 23 us; skew 500 ppm
I20240501 19:56:51.237403 3621 webserver.cc:460] Webserver started at http://127.30.1.65:33291/ using document root <none> and password file <none>
I20240501 19:56:51.237888 3621 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:51.237957 3621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:51.240790 3621 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:51.243140 3637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:51.244032 3621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:51.244125 3621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "00bf2be91e8f4a1a8b6fe07c478bf543"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-htmr"
I20240501 19:56:51.244475 3621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:51.256494 3621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:51.257017 3621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:51.257164 3621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:51.257642 3621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:51.258669 3644 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240501 19:56:51.267851 3621 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240501 19:56:51.267916 3621 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:51.267969 3621 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240501 19:56:51.269907 3644 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.273614 3621 ts_tablet_manager.cc:590] Registered 12 tablets
I20240501 19:56:51.273658 3621 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.002s sys 0.003s
I20240501 19:56:51.285786 3644 log.cc:824] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:51.287684 3621 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.65:43707
I20240501 19:56:51.287752 3754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.65:43707 every 8 connection(s)
I20240501 19:56:51.288705 3621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:51.289448 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 3621
I20240501 19:56:51.297533 3755 heartbeater.cc:347] Connected to a master server at 127.30.1.126:46341
I20240501 19:56:51.297762 3755 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:51.298224 3755 heartbeater.cc:511] Master 127.30.1.126:46341 requested a full tablet report, sending...
I20240501 19:56:51.299898 30765 ts_manager.cc:194] Re-registered known tserver with Master: 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707)
I20240501 19:56:51.301894 30765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.65:42641
I20240501 19:56:51.317860 3644 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.318256 3644 tablet_bootstrap.cc:492] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.319432 3644 ts_tablet_manager.cc:1367] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.050s user 0.037s sys 0.009s
I20240501 19:56:51.321683 3644 raft_consensus.cc:348] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.322046 3644 raft_consensus.cc:729] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.322467 3644 consensus_queue.cc:260] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.322979 3755 heartbeater.cc:503] Master 127.30.1.126:46341 was elected leader, sending a full tablet report...
I20240501 19:56:51.323237 3644 ts_tablet_manager.cc:1398] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 19:56:51.323473 3644 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.353482 3644 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.353806 3644 tablet_bootstrap.cc:492] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.354741 3644 ts_tablet_manager.cc:1367] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.003s
I20240501 19:56:51.355199 3644 raft_consensus.cc:348] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.355327 3644 raft_consensus.cc:729] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.355453 3644 consensus_queue.cc:260] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.355682 3644 ts_tablet_manager.cc:1398] T 9de501bd2ff545f9842d386a2ea8fba8 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.355881 3644 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.384703 3644 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.385084 3644 tablet_bootstrap.cc:492] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.386195 3644 ts_tablet_manager.cc:1367] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.005s
I20240501 19:56:51.386638 3644 raft_consensus.cc:348] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.386795 3644 raft_consensus.cc:729] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.386965 3644 consensus_queue.cc:260] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.387230 3644 ts_tablet_manager.cc:1398] T bb93992159a3461b945d31da3c9bb43f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:51.387415 3644 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.419917 3644 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.420267 3644 tablet_bootstrap.cc:492] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.421379 3644 ts_tablet_manager.cc:1367] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.009s
I20240501 19:56:51.421787 3644 raft_consensus.cc:348] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:51.421944 3644 raft_consensus.cc:729] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.422169 3644 consensus_queue.cc:260] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:51.422453 3644 ts_tablet_manager.cc:1398] T dbe3171686254a27a3bbe08786fa7503 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.422649 3644 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.432214 3685 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9033765edb40b1ab49d4766eddf1ff with delete_type TABLET_DATA_TOMBSTONED (TS 00bf2be91e8f4a1a8b6fe07c478bf543 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57198
I20240501 19:56:51.432669 3768 tablet_replica.cc:330] stopping tablet replica
I20240501 19:56:51.432955 3768 raft_consensus.cc:2229] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.433161 3768 raft_consensus.cc:2258] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.433899 3768 ts_tablet_manager.cc:1875] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.436791 3768 ts_tablet_manager.cc:1888] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:51.436861 3768 log.cc:1196] T ac9033765edb40b1ab49d4766eddf1ff P 00bf2be91e8f4a1a8b6fe07c478bf543: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac9033765edb40b1ab49d4766eddf1ff
I20240501 19:56:51.437384 30753 catalog_manager.cc:4830] TS 00bf2be91e8f4a1a8b6fe07c478bf543 (127.30.1.65:43707): tablet ac9033765edb40b1ab49d4766eddf1ff (table rebalance_test_table_4 [id=824079444b164d49be1f7acbc0ea02fe]) successfully deleted
I20240501 19:56:51.450592 3644 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.450848 3644 tablet_bootstrap.cc:492] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.451800 3644 ts_tablet_manager.cc:1367] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.012s
I20240501 19:56:51.452221 3644 raft_consensus.cc:348] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.452308 3644 raft_consensus.cc:729] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.452436 3644 consensus_queue.cc:260] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.452661 3644 ts_tablet_manager.cc:1398] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 19:56:51.452798 3644 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.477885 3644 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.478199 3644 tablet_bootstrap.cc:492] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.479097 3644 ts_tablet_manager.cc:1367] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240501 19:56:51.479468 3644 raft_consensus.cc:348] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:51.479557 3644 raft_consensus.cc:729] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.479679 3644 consensus_queue.cc:260] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } }
I20240501 19:56:51.479928 3644 ts_tablet_manager.cc:1398] T 6c061670c33145f389f4767a355537cf P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.480055 3644 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.508299 3644 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.508666 3644 tablet_bootstrap.cc:492] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.509698 3644 ts_tablet_manager.cc:1367] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.007s
I20240501 19:56:51.510218 3644 raft_consensus.cc:348] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.510342 3644 raft_consensus.cc:729] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.510502 3644 consensus_queue.cc:260] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.510798 3644 ts_tablet_manager.cc:1398] T 4f614c0ec5274c31a99bff11fe40307f P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.510980 3644 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.521023 3685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.526254 31021 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.528507 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:51.529968 32537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:51.537993 32406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.538237 32275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.539232 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.549612 3644 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.549753 3696 raft_consensus.cc:3037] T 2edc89ebb6984377ab459e5e219a4fab P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:51.549916 3644 tablet_bootstrap.cc:492] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.550956 3644 ts_tablet_manager.cc:1367] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.000s
I20240501 19:56:51.551317 3644 raft_consensus.cc:348] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.551419 3644 raft_consensus.cc:729] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.551578 3644 consensus_queue.cc:260] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.551895 3644 ts_tablet_manager.cc:1398] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:51.552019 3644 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.574044 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.584520 3644 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.584865 3644 tablet_bootstrap.cc:492] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.586038 3644 ts_tablet_manager.cc:1367] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.010s
I20240501 19:56:51.586380 3644 raft_consensus.cc:348] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:51.586485 3644 raft_consensus.cc:729] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.586628 3644 consensus_queue.cc:260] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } } peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } }
I20240501 19:56:51.586933 3644 ts_tablet_manager.cc:1398] T 33ae5913b9cc43c1a82b2d03646983c5 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:51.587069 3644 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.617491 3644 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.617772 3644 tablet_bootstrap.cc:492] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.618747 3644 ts_tablet_manager.cc:1367] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.001s
I20240501 19:56:51.619151 3644 raft_consensus.cc:348] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:51.619263 3644 raft_consensus.cc:729] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.619453 3644 consensus_queue.cc:260] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "1cd4016c2d8f4feb9fb1fd5a8be29fca" member_type: NON_VOTER last_known_addr { host: "127.30.1.70" port: 39065 } attrs { promote: true } }
I20240501 19:56:51.619727 3644 ts_tablet_manager.cc:1398] T 33c7d50279364b53953d0a0d45edfb9a P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:51.619913 3644 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.647665 3644 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.648017 3644 tablet_bootstrap.cc:492] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.649199 3644 ts_tablet_manager.cc:1367] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.008s
I20240501 19:56:51.649641 3644 raft_consensus.cc:348] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.649801 3644 raft_consensus.cc:729] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.649966 3644 consensus_queue.cc:260] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d66d1d57441db81def86faef0eb4a" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 36463 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.650629 3644 ts_tablet_manager.cc:1398] T 0684929c231d45a086b6356a9e6922ec P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:51.650803 3644 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap starting.
I20240501 19:56:51.657480 3696 raft_consensus.cc:3037] T b2ccd7f3fea4473497944de737911828 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:51.672323 3685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.675921 31021 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.680055 31283 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.683581 3644 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.683625 32406 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:51.683917 3644 tablet_bootstrap.cc:492] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Bootstrap complete.
I20240501 19:56:51.685067 32668 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.685494 3644 ts_tablet_manager.cc:1367] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20240501 19:56:51.685866 3644 raft_consensus.cc:348] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.686012 3644 raft_consensus.cc:729] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bf2be91e8f4a1a8b6fe07c478bf543, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.686172 3644 consensus_queue.cc:260] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d51481decd4d06964f09c66f981e09" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 37547 } } peers { permanent_uuid: "5530341eedde430fbacb4c3ff815159c" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 37085 } } peers { permanent_uuid: "00bf2be91e8f4a1a8b6fe07c478bf543" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 43707 } }
I20240501 19:56:51.686488 3644 ts_tablet_manager.cc:1398] T 274800fe27c245139788f075eca1e812 P 00bf2be91e8f4a1a8b6fe07c478bf543: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.690073 32537 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:51.704284 31152 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:51.715046 32275 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:51.745278 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 3621
I20240501 19:56:51.751399 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 30959
I20240501 19:56:51.758002 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31090
I20240501 19:56:51.764149 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 31221
I20240501 19:56:51.769941 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 32213
I20240501 19:56:51.775545 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 32344
I20240501 19:56:51.781189 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 32474
I20240501 19:56:51.786554 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 32606
I20240501 19:56:51.792093 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 30737
2024-05-01T19:56:51Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (54019 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (54019 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240501 19:56:51.826581 30725 test_util.cc:268] Using random seed: -482409582
I20240501 19:56:51.827281 30725 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:51.827320 30725 ts_itest-base.cc:113] --------------
I20240501 19:56:51.827342 30725 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:56:51.827356 30725 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:51.827383 30725 ts_itest-base.cc:116] --------------
2024-05-01T19:56:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:51Z Disabled control of system clock
I20240501 19:56:51.834275 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:43499
--webserver_interface=127.30.1.126
--webserver_port=0
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.1.126:43499
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 19:56:51.930601 3868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:51.930783 3868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:51.930845 3868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:51.933686 3868 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:51.933741 3868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:51.933769 3868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:51.933808 3868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:51.933836 3868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:51.937007 3868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.1.126:43499
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.1.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:51.937685 3868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:51.938556 3868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:51.944865 3874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:51.945451 3876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:51.946156 3873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:51.946883 3868 server_base.cc:1034] running on GCE node
I20240501 19:56:51.947261 3868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:51.947991 3868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:51.949132 3868 hybrid_clock.cc:648] HybridClock initialized: now 1714593411949123 us; error 23 us; skew 500 ppm
I20240501 19:56:51.950721 3868 webserver.cc:460] Webserver started at http://127.30.1.126:38459/ using document root <none> and password file <none>
I20240501 19:56:51.951180 3868 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:51.951238 3868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:51.951390 3868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:51.952767 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733"
format_stamp: "Formatted at 2024-05-01 19:56:51 on dist-test-slave-htmr"
I20240501 19:56:51.953219 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733"
format_stamp: "Formatted at 2024-05-01 19:56:51 on dist-test-slave-htmr"
I20240501 19:56:51.956008 3868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:51.957854 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:51.958681 3868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:51.958787 3868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733"
format_stamp: "Formatted at 2024-05-01 19:56:51 on dist-test-slave-htmr"
I20240501 19:56:51.958880 3868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:51.968130 3868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:51.968593 3868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:51.968750 3868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:51.974665 3868 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:43499
I20240501 19:56:51.974684 3934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:43499 every 8 connection(s)
I20240501 19:56:51.975445 3868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:51.977658 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.978554 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 3868
I20240501 19:56:51.978688 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:51.982471 3935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733: Bootstrap starting.
I20240501 19:56:51.984349 3935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:51.985019 3935 log.cc:824] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:51.986445 3935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733: No bootstrap required, opened a new log
I20240501 19:56:51.988826 3935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } }
I20240501 19:56:51.989027 3935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:51.989069 3935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1b36bdcd0746fdab4a8e2d2d9f2733, State: Initialized, Role: FOLLOWER
I20240501 19:56:51.989524 3935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } }
I20240501 19:56:51.989660 3935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:51.989737 3935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:51.989854 3935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:51.990449 3935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } }
I20240501 19:56:51.990713 3935 leader_election.cc:302] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f1b36bdcd0746fdab4a8e2d2d9f2733; no voters:
I20240501 19:56:51.990937 3935 leader_election.cc:288] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:51.991026 3940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:51.991264 3940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1b36bdcd0746fdab4a8e2d2d9f2733, State: Running, Role: LEADER
I20240501 19:56:51.991849 3935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:51.991858 3940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } }
I20240501 19:56:51.993094 3941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } } }
I20240501 19:56:51.993221 3941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:51.993530 3942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f1b36bdcd0746fdab4a8e2d2d9f2733. Latest consensus state: current_term: 1 leader_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1b36bdcd0746fdab4a8e2d2d9f2733" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 43499 } } }
I20240501 19:56:51.993584 3951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:51.993640 3942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:51.996157 3951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:52.000200 3951 catalog_manager.cc:1347] Generated new cluster ID: d06223efe8ea409ea4618a61152f60d5
I20240501 19:56:52.000257 3951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:52.015136 3951 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:52.015868 3951 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:52.021427 3951 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5f1b36bdcd0746fdab4a8e2d2d9f2733: Generated new TSK 0
I20240501 19:56:52.021942 3951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:52.025688 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.65:0
--local_ip_for_outbound_sockets=127.30.1.65
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:52.126367 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:52.126533 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:52.126574 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:52.129577 3959 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:52.129628 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:52.129707 3959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.65
I20240501 19:56:52.133944 3959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--never_fsync=true
--heap_profile_path=/tmp/kudu.3959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.65
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:52.134554 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:52.135385 3959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.143136 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.143787 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.144440 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.145113 3959 server_base.cc:1034] running on GCE node
I20240501 19:56:52.145427 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:52.145901 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:52.147071 3959 hybrid_clock.cc:648] HybridClock initialized: now 1714593412147051 us; error 44 us; skew 500 ppm
I20240501 19:56:52.148746 3959 webserver.cc:460] Webserver started at http://127.30.1.65:35669/ using document root <none> and password file <none>
I20240501 19:56:52.149217 3959 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.149273 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.149418 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.150854 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7342f4c74fa842509be00e8c334dbbc9"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.151309 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7342f4c74fa842509be00e8c334dbbc9"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.154196 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:52.156147 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.157059 3959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:52.157150 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7342f4c74fa842509be00e8c334dbbc9"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.157223 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.169498 3959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.169948 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:52.170105 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.170542 3959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.171308 3959 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.171353 3959 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.171419 3959 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.171449 3959 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.179219 3959 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.65:42857
I20240501 19:56:52.179261 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.65:42857 every 8 connection(s)
I20240501 19:56:52.179996 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:52.181097 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 3959
I20240501 19:56:52.181200 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:52.183207 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.66:0
--local_ip_for_outbound_sockets=127.30.1.66
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:52.188393 4087 heartbeater.cc:347] Connected to a master server at 127.30.1.126:43499
I20240501 19:56:52.188580 4087 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.188987 4087 heartbeater.cc:511] Master 127.30.1.126:43499 requested a full tablet report, sending...
I20240501 19:56:52.190451 3899 ts_manager.cc:194] Registered new tserver with Master: 7342f4c74fa842509be00e8c334dbbc9 (127.30.1.65:42857)
I20240501 19:56:52.192093 3899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.65:52009
W20240501 19:56:52.285527 4090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:52.285702 4090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:52.285766 4090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:52.289618 4090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:52.289695 4090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:52.289798 4090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.66
I20240501 19:56:52.294139 4090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--never_fsync=true
--heap_profile_path=/tmp/kudu.4090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.66
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:52.294986 4090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:52.296061 4090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.303308 4095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.304062 4096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.304612 4098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.305311 4090 server_base.cc:1034] running on GCE node
I20240501 19:56:52.305613 4090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:52.306054 4090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:52.307225 4090 hybrid_clock.cc:648] HybridClock initialized: now 1714593412307200 us; error 38 us; skew 500 ppm
I20240501 19:56:52.308820 4090 webserver.cc:460] Webserver started at http://127.30.1.66:38475/ using document root <none> and password file <none>
I20240501 19:56:52.309324 4090 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.309376 4090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.309516 4090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.310861 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f3377bcee9c34b00871079d8fb9e2e53"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.311287 4090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f3377bcee9c34b00871079d8fb9e2e53"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.314188 4090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 19:56:52.316016 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.316810 4090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:52.316921 4090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f3377bcee9c34b00871079d8fb9e2e53"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.316996 4090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.332033 4090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.332501 4090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:52.332640 4090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.333101 4090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.333807 4090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.333853 4090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.333901 4090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.333927 4090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.341408 4090 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.66:40295
I20240501 19:56:52.341428 4217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.66:40295 every 8 connection(s)
I20240501 19:56:52.342164 4090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:52.349558 4218 heartbeater.cc:347] Connected to a master server at 127.30.1.126:43499
I20240501 19:56:52.349726 4218 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.349972 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4090
I20240501 19:56:52.350061 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:52.350088 4218 heartbeater.cc:511] Master 127.30.1.126:43499 requested a full tablet report, sending...
I20240501 19:56:52.350911 3899 ts_manager.cc:194] Registered new tserver with Master: f3377bcee9c34b00871079d8fb9e2e53 (127.30.1.66:40295)
I20240501 19:56:52.351516 3899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.66:52015
I20240501 19:56:52.352066 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.67:0
--local_ip_for_outbound_sockets=127.30.1.67
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:52.447989 4221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:52.448163 4221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:52.448204 4221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:52.451094 4221 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:52.451164 4221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:52.451226 4221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.67
I20240501 19:56:52.454447 4221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--never_fsync=true
--heap_profile_path=/tmp/kudu.4221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.67
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:52.455049 4221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:52.455946 4221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.463132 4227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.463841 4226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.464509 4229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.465191 4221 server_base.cc:1034] running on GCE node
I20240501 19:56:52.465529 4221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:52.465960 4221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:52.467103 4221 hybrid_clock.cc:648] HybridClock initialized: now 1714593412467094 us; error 19 us; skew 500 ppm
I20240501 19:56:52.468647 4221 webserver.cc:460] Webserver started at http://127.30.1.67:42995/ using document root <none> and password file <none>
I20240501 19:56:52.469158 4221 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.469215 4221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.469362 4221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.470736 4221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c7120a526b134696ba1ac2489cee6b7b"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.471135 4221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c7120a526b134696ba1ac2489cee6b7b"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.474427 4221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 19:56:52.476518 4235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.477371 4221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:52.477455 4221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c7120a526b134696ba1ac2489cee6b7b"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.477553 4221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.502501 4221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.502950 4221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:52.503082 4221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.503491 4221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.504155 4221 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.504204 4221 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.504240 4221 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.504266 4221 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.511793 4221 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.67:44273
I20240501 19:56:52.511819 4348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.67:44273 every 8 connection(s)
I20240501 19:56:52.512540 4221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:52.517016 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4221
I20240501 19:56:52.517107 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:52.519112 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.68:0
--local_ip_for_outbound_sockets=127.30.1.68
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:52.519874 4349 heartbeater.cc:347] Connected to a master server at 127.30.1.126:43499
I20240501 19:56:52.520042 4349 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.520371 4349 heartbeater.cc:511] Master 127.30.1.126:43499 requested a full tablet report, sending...
I20240501 19:56:52.521266 3899 ts_manager.cc:194] Registered new tserver with Master: c7120a526b134696ba1ac2489cee6b7b (127.30.1.67:44273)
I20240501 19:56:52.521818 3899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.67:51387
W20240501 19:56:52.614293 4353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:52.614461 4353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:52.614523 4353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:52.617298 4353 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:52.617357 4353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:52.617444 4353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.68
I20240501 19:56:52.620707 4353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--never_fsync=true
--heap_profile_path=/tmp/kudu.4353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.68
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:52.621413 4353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:52.622316 4353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.629518 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.630180 4359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.630863 4358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.631610 4353 server_base.cc:1034] running on GCE node
I20240501 19:56:52.631958 4353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:52.632431 4353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:52.633543 4353 hybrid_clock.cc:648] HybridClock initialized: now 1714593412633529 us; error 40 us; skew 500 ppm
I20240501 19:56:52.635027 4353 webserver.cc:460] Webserver started at http://127.30.1.68:35017/ using document root <none> and password file <none>
I20240501 19:56:52.635444 4353 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.635493 4353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.635628 4353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.637005 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "050f92f72df84e16a2fd8e4982ba35a6"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.637427 4353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "050f92f72df84e16a2fd8e4982ba35a6"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.640228 4353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:52.642096 4367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.642895 4353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:52.642987 4353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "050f92f72df84e16a2fd8e4982ba35a6"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.643060 4353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.652981 4353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.653407 4353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:52.653532 4353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.653913 4353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.654577 4353 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.654636 4353 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.654687 4353 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.654721 4353 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.662636 4353 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.68:45211
I20240501 19:56:52.662679 4481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.68:45211 every 8 connection(s)
I20240501 19:56:52.663372 4353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:52.664016 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4353
I20240501 19:56:52.664113 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:52.666010 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:0
--local_ip_for_outbound_sockets=127.30.1.69
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--builtin_ntp_servers=127.30.1.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:52.671110 4482 heartbeater.cc:347] Connected to a master server at 127.30.1.126:43499
I20240501 19:56:52.671291 4482 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.671654 4482 heartbeater.cc:511] Master 127.30.1.126:43499 requested a full tablet report, sending...
I20240501 19:56:52.672484 3899 ts_manager.cc:194] Registered new tserver with Master: 050f92f72df84e16a2fd8e4982ba35a6 (127.30.1.68:45211)
I20240501 19:56:52.673023 3899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.68:37703
W20240501 19:56:52.762447 4485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:52.762631 4485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:52.762689 4485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:52.765579 4485 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:52.765630 4485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:52.765723 4485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:56:52.769050 4485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:43499
--never_fsync=true
--heap_profile_path=/tmp/kudu.4485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:52.769696 4485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:52.770581 4485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.778082 4490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.778846 4491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.779585 4493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.780349 4485 server_base.cc:1034] running on GCE node
I20240501 19:56:52.780664 4485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:52.781196 4485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:52.782331 4485 hybrid_clock.cc:648] HybridClock initialized: now 1714593412782317 us; error 28 us; skew 500 ppm
I20240501 19:56:52.783897 4485 webserver.cc:460] Webserver started at http://127.30.1.69:44625/ using document root <none> and password file <none>
I20240501 19:56:52.784368 4485 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.784420 4485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.784582 4485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.785998 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "36127b23a780430e8513cabe76aa259d"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.786419 4485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "36127b23a780430e8513cabe76aa259d"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.789255 4485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:52.791234 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.792127 4485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:52.792213 4485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "36127b23a780430e8513cabe76aa259d"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-htmr"
I20240501 19:56:52.792351 4485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.801100 4485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.801525 4485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:52.801681 4485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.802093 4485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.802824 4485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.802867 4485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.802907 4485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.802940 4485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.810510 4485 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:35743
I20240501 19:56:52.810539 4612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:35743 every 8 connection(s)
I20240501 19:56:52.811282 4485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:52.818761 4613 heartbeater.cc:347] Connected to a master server at 127.30.1.126:43499
I20240501 19:56:52.818923 4613 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.819249 4613 heartbeater.cc:511] Master 127.30.1.126:43499 requested a full tablet report, sending...
I20240501 19:56:52.820092 3899 ts_manager.cc:194] Registered new tserver with Master: 36127b23a780430e8513cabe76aa259d (127.30.1.69:35743)
I20240501 19:56:52.820611 3899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:51753
I20240501 19:56:52.820638 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4485
I20240501 19:56:52.820727 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:52.822691 30725 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:56:52.830438 3899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58474:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:56:52.846089 4021 tablet_service.cc:1467] Processing CreateTablet for tablet f4d37f84ec904056b0a723b9034daaf2 (DEFAULT_TABLE table=TestTable [id=90a3c5a074204128a94a832d6e19386c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.846207 4415 tablet_service.cc:1467] Processing CreateTablet for tablet f4d37f84ec904056b0a723b9034daaf2 (DEFAULT_TABLE table=TestTable [id=90a3c5a074204128a94a832d6e19386c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.846964 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d37f84ec904056b0a723b9034daaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.846971 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d37f84ec904056b0a723b9034daaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.846966 4283 tablet_service.cc:1467] Processing CreateTablet for tablet f4d37f84ec904056b0a723b9034daaf2 (DEFAULT_TABLE table=TestTable [id=90a3c5a074204128a94a832d6e19386c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:52.847664 4283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d37f84ec904056b0a723b9034daaf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.851464 4630 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Bootstrap starting.
I20240501 19:56:52.852516 4631 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Bootstrap starting.
I20240501 19:56:52.852845 4632 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Bootstrap starting.
I20240501 19:56:52.853575 4630 tablet_bootstrap.cc:654] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.854269 4630 log.cc:824] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:52.854387 4632 tablet_bootstrap.cc:654] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.854410 4631 tablet_bootstrap.cc:654] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.855046 4631 log.cc:824] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:52.855053 4632 log.cc:824] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:52.855865 4630 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: No bootstrap required, opened a new log
I20240501 19:56:52.856005 4630 ts_tablet_manager.cc:1367] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:52.856516 4631 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: No bootstrap required, opened a new log
I20240501 19:56:52.856516 4632 tablet_bootstrap.cc:492] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: No bootstrap required, opened a new log
I20240501 19:56:52.856667 4632 ts_tablet_manager.cc:1367] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:52.856667 4631 ts_tablet_manager.cc:1367] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:52.858691 4630 raft_consensus.cc:348] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.858860 4630 raft_consensus.cc:374] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.858935 4630 raft_consensus.cc:729] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050f92f72df84e16a2fd8e4982ba35a6, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.859309 4630 consensus_queue.cc:260] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.859854 4482 heartbeater.cc:503] Master 127.30.1.126:43499 was elected leader, sending a full tablet report...
I20240501 19:56:52.860078 4630 ts_tablet_manager.cc:1398] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:52.860128 4632 raft_consensus.cc:348] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.860325 4632 raft_consensus.cc:374] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.860385 4632 raft_consensus.cc:729] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7120a526b134696ba1ac2489cee6b7b, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.860128 4631 raft_consensus.cc:348] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.860850 4632 consensus_queue.cc:260] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.861039 4631 raft_consensus.cc:374] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.861152 4631 raft_consensus.cc:729] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7342f4c74fa842509be00e8c334dbbc9, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.861521 4349 heartbeater.cc:503] Master 127.30.1.126:43499 was elected leader, sending a full tablet report...
I20240501 19:56:52.861621 4631 consensus_queue.cc:260] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:52.861750 4632 ts_tablet_manager.cc:1398] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:56:52.862473 4631 ts_tablet_manager.cc:1398] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:52.862798 4087 heartbeater.cc:503] Master 127.30.1.126:43499 was elected leader, sending a full tablet report...
W20240501 19:56:52.914127 4483 tablet.cc:2369] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:52.931174 4088 tablet.cc:2369] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:53.013491 4350 tablet.cc:2369] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:53.039191 4638 raft_consensus.cc:482] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:53.039353 4638 raft_consensus.cc:504] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:53.040328 4638 leader_election.cc:288] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 050f92f72df84e16a2fd8e4982ba35a6 (127.30.1.68:45211), c7120a526b134696ba1ac2489cee6b7b (127.30.1.67:44273)
I20240501 19:56:53.043805 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d37f84ec904056b0a723b9034daaf2" candidate_uuid: "7342f4c74fa842509be00e8c334dbbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f92f72df84e16a2fd8e4982ba35a6" is_pre_election: true
I20240501 19:56:53.043818 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d37f84ec904056b0a723b9034daaf2" candidate_uuid: "7342f4c74fa842509be00e8c334dbbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7120a526b134696ba1ac2489cee6b7b" is_pre_election: true
I20240501 19:56:53.044042 4435 raft_consensus.cc:2454] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7342f4c74fa842509be00e8c334dbbc9 in term 0.
I20240501 19:56:53.044049 4303 raft_consensus.cc:2454] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7342f4c74fa842509be00e8c334dbbc9 in term 0.
I20240501 19:56:53.044417 3974 leader_election.cc:302] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f92f72df84e16a2fd8e4982ba35a6, 7342f4c74fa842509be00e8c334dbbc9; no voters:
I20240501 19:56:53.044623 4638 raft_consensus.cc:2781] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:53.044692 4638 raft_consensus.cc:482] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:53.044731 4638 raft_consensus.cc:3037] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.045558 4638 raft_consensus.cc:504] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:53.045838 4638 leader_election.cc:288] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [CANDIDATE]: Term 1 election: Requested vote from peers 050f92f72df84e16a2fd8e4982ba35a6 (127.30.1.68:45211), c7120a526b134696ba1ac2489cee6b7b (127.30.1.67:44273)
I20240501 19:56:53.046133 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d37f84ec904056b0a723b9034daaf2" candidate_uuid: "7342f4c74fa842509be00e8c334dbbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "050f92f72df84e16a2fd8e4982ba35a6"
I20240501 19:56:53.046234 4435 raft_consensus.cc:3037] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.046312 4303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d37f84ec904056b0a723b9034daaf2" candidate_uuid: "7342f4c74fa842509be00e8c334dbbc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7120a526b134696ba1ac2489cee6b7b"
I20240501 19:56:53.046425 4303 raft_consensus.cc:3037] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:53.047114 4435 raft_consensus.cc:2454] T f4d37f84ec904056b0a723b9034daaf2 P 050f92f72df84e16a2fd8e4982ba35a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7342f4c74fa842509be00e8c334dbbc9 in term 1.
I20240501 19:56:53.047346 3974 leader_election.cc:302] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 050f92f72df84e16a2fd8e4982ba35a6, 7342f4c74fa842509be00e8c334dbbc9; no voters:
I20240501 19:56:53.047375 4303 raft_consensus.cc:2454] T f4d37f84ec904056b0a723b9034daaf2 P c7120a526b134696ba1ac2489cee6b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7342f4c74fa842509be00e8c334dbbc9 in term 1.
I20240501 19:56:53.047508 4638 raft_consensus.cc:2781] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:53.047722 4638 raft_consensus.cc:686] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7342f4c74fa842509be00e8c334dbbc9, State: Running, Role: LEADER
I20240501 19:56:53.047996 4638 consensus_queue.cc:237] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } }
I20240501 19:56:53.050444 3899 catalog_manager.cc:5484] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7342f4c74fa842509be00e8c334dbbc9 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "7342f4c74fa842509be00e8c334dbbc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7342f4c74fa842509be00e8c334dbbc9" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 42857 } } peers { permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 } } peers { permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 } } }
I20240501 19:56:53.071796 30725 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:56:53.073359 30725 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7342f4c74fa842509be00e8c334dbbc9 to finish bootstrapping
I20240501 19:56:53.078079 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f3377bcee9c34b00871079d8fb9e2e53 to finish bootstrapping
I20240501 19:56:53.082983 30725 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c7120a526b134696ba1ac2489cee6b7b to finish bootstrapping
I20240501 19:56:53.087071 30725 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 050f92f72df84e16a2fd8e4982ba35a6 to finish bootstrapping
I20240501 19:56:53.090895 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 36127b23a780430e8513cabe76aa259d to finish bootstrapping
I20240501 19:56:53.275674 4547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.276592 4283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.278913 4415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.288199 4021 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:53.292415 4152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.353175 4218 heartbeater.cc:503] Master 127.30.1.126:43499 was elected leader, sending a full tablet report...
I20240501 19:56:53.366781 4415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.367054 4021 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:53.368476 4283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.376335 4152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.376591 4547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.434208 4638 consensus_queue.cc:1035] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050f92f72df84e16a2fd8e4982ba35a6" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.438863 4641 consensus_queue.cc:1035] T f4d37f84ec904056b0a723b9034daaf2 P 7342f4c74fa842509be00e8c334dbbc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7120a526b134696ba1ac2489cee6b7b" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 44273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:53.570650 4415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.570724 4547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.571167 4021 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:53.573002 4152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.574331 4283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.660331 4415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.660676 4547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.660929 4283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.662415 4152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.663741 4021 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:53.755832 4415 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.757505 4021 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:53.757706 4547 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.757776 4152 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.759459 4283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.797251 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 3959
I20240501 19:56:53.803297 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4090
I20240501 19:56:53.808404 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4221
I20240501 19:56:53.813750 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4353
I20240501 19:56:53.820042 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4485
I20240501 19:56:53.824296 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 3868
2024-05-01T19:56:53Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2019 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2019 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240501 19:56:53.846571 30725 test_util.cc:268] Using random seed: -480389592
I20240501 19:56:53.847244 30725 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:53.847275 30725 ts_itest-base.cc:113] --------------
I20240501 19:56:53.847287 30725 ts_itest-base.cc:114] 7 tablet servers
I20240501 19:56:53.847297 30725 ts_itest-base.cc:115] 2 replicas per TS
I20240501 19:56:53.847321 30725 ts_itest-base.cc:116] --------------
2024-05-01T19:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:53Z Disabled control of system clock
I20240501 19:56:53.854660 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:44833
--webserver_interface=127.30.1.126
--webserver_port=0
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.1.126:44833
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 19:56:53.949623 4824 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 19:56:53.949806 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:53.949831 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:53.949862 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:53.952695 4824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:53.952754 4824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:53.952780 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:53.952800 4824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:53.952828 4824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:53.956212 4824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.1.126:44833
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.1.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:53.956874 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:53.957810 4824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:53.964789 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:53.965458 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:53.966166 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:53.966893 4824 server_base.cc:1034] running on GCE node
I20240501 19:56:53.967289 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:53.968045 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:53.969185 4824 hybrid_clock.cc:648] HybridClock initialized: now 1714593413969177 us; error 23 us; skew 500 ppm
I20240501 19:56:53.970827 4824 webserver.cc:460] Webserver started at http://127.30.1.126:36153/ using document root <none> and password file <none>
I20240501 19:56:53.971311 4824 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:53.971362 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:53.971529 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:53.973035 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c129cc94b2be4bc1bb1290501c6a5390"
format_stamp: "Formatted at 2024-05-01 19:56:53 on dist-test-slave-htmr"
I20240501 19:56:53.973476 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c129cc94b2be4bc1bb1290501c6a5390"
format_stamp: "Formatted at 2024-05-01 19:56:53 on dist-test-slave-htmr"
I20240501 19:56:53.976353 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:53.978332 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:53.979238 4824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:53.979348 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c129cc94b2be4bc1bb1290501c6a5390"
format_stamp: "Formatted at 2024-05-01 19:56:53 on dist-test-slave-htmr"
I20240501 19:56:53.979425 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:53.994206 4824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:53.994727 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:53.994869 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.000762 4824 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:44833
I20240501 19:56:54.000783 4891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:44833 every 8 connection(s)
I20240501 19:56:54.001602 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:54.003816 4892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.008220 4892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390: Bootstrap starting.
I20240501 19:56:54.009071 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4824
I20240501 19:56:54.009171 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:54.010430 4892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:54.011049 4892 log.cc:824] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:54.012530 4892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390: No bootstrap required, opened a new log
I20240501 19:56:54.014982 4892 raft_consensus.cc:348] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } }
I20240501 19:56:54.015136 4892 raft_consensus.cc:374] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:54.015173 4892 raft_consensus.cc:729] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c129cc94b2be4bc1bb1290501c6a5390, State: Initialized, Role: FOLLOWER
I20240501 19:56:54.015630 4892 consensus_queue.cc:260] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } }
I20240501 19:56:54.015803 4892 raft_consensus.cc:388] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:54.015868 4892 raft_consensus.cc:482] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:54.015969 4892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:54.016707 4892 raft_consensus.cc:504] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } }
I20240501 19:56:54.017024 4892 leader_election.cc:302] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c129cc94b2be4bc1bb1290501c6a5390; no voters:
I20240501 19:56:54.017283 4892 leader_election.cc:288] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:54.017616 4897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:54.017799 4897 raft_consensus.cc:686] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [term 1 LEADER]: Becoming Leader. State: Replica: c129cc94b2be4bc1bb1290501c6a5390, State: Running, Role: LEADER
I20240501 19:56:54.018072 4897 consensus_queue.cc:237] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } }
I20240501 19:56:54.018275 4892 sys_catalog.cc:564] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:54.019338 4898 sys_catalog.cc:455] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c129cc94b2be4bc1bb1290501c6a5390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } } }
I20240501 19:56:54.019457 4898 sys_catalog.cc:458] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:54.019619 4899 sys_catalog.cc:455] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c129cc94b2be4bc1bb1290501c6a5390. Latest consensus state: current_term: 1 leader_uuid: "c129cc94b2be4bc1bb1290501c6a5390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129cc94b2be4bc1bb1290501c6a5390" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 44833 } } }
I20240501 19:56:54.019827 4899 sys_catalog.cc:458] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:54.020042 4907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:54.022322 4907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:54.026582 4907 catalog_manager.cc:1347] Generated new cluster ID: f4f34d730fda406b9c59e482ec473a03
I20240501 19:56:54.026631 4907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:54.042032 4907 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:54.042730 4907 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:54.052053 4907 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c129cc94b2be4bc1bb1290501c6a5390: Generated new TSK 0
I20240501 19:56:54.052613 4907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:54.056342 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.65:0
--local_ip_for_outbound_sockets=127.30.1.65
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:56:54.169497 4916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.169665 4916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.169731 4916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.172695 4916 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.172760 4916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.172852 4916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.65
I20240501 19:56:54.176223 4916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.4916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.65
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.176936 4916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.177878 4916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:54.185523 4924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.186043 4922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.186818 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:54.187553 4916 server_base.cc:1034] running on GCE node
I20240501 19:56:54.187873 4916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:54.188349 4916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:54.189513 4916 hybrid_clock.cc:648] HybridClock initialized: now 1714593414189471 us; error 61 us; skew 500 ppm
I20240501 19:56:54.191208 4916 webserver.cc:460] Webserver started at http://127.30.1.65:44965/ using document root <none> and password file <none>
I20240501 19:56:54.191684 4916 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:54.191735 4916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:54.191888 4916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:54.193459 4916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b051b1f42cb443e58813b92c5ae4a62e"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.193858 4916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b051b1f42cb443e58813b92c5ae4a62e"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.197021 4916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:54.198897 4930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.199723 4916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:54.199815 4916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b051b1f42cb443e58813b92c5ae4a62e"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.199901 4916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:54.224925 4916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:54.225443 4916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:54.225579 4916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.226038 4916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:54.226795 4916 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:54.226841 4916 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.226882 4916 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:54.226914 4916 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.234937 4916 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.65:32835
I20240501 19:56:54.234966 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.65:32835 every 8 connection(s)
I20240501 19:56:54.235726 4916 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:54.241832 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 4916
I20240501 19:56:54.241947 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:54.243666 5045 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:54.243851 5045 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:54.244010 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.66:0
--local_ip_for_outbound_sockets=127.30.1.66
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:54.244258 5045 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:54.245559 4855 ts_manager.cc:194] Registered new tserver with Master: b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:56:54.247090 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.65:42541
W20240501 19:56:54.340691 5048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.340843 5048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.340881 5048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.343667 5048 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.343719 5048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.343777 5048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.66
I20240501 19:56:54.347028 5048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.66
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.347703 5048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.348592 5048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:54.355638 5053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.356371 5054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.357020 5056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:54.357733 5048 server_base.cc:1034] running on GCE node
I20240501 19:56:54.358042 5048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:54.358500 5048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:54.359642 5048 hybrid_clock.cc:648] HybridClock initialized: now 1714593414359629 us; error 27 us; skew 500 ppm
I20240501 19:56:54.361177 5048 webserver.cc:460] Webserver started at http://127.30.1.66:45937/ using document root <none> and password file <none>
I20240501 19:56:54.361629 5048 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:54.361685 5048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:54.361845 5048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:54.363188 5048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.363588 5048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.366413 5048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:54.368216 5062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.369135 5048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:54.369226 5048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.369300 5048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:54.384663 5048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:54.385186 5048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:54.385321 5048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.385728 5048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:54.386449 5048 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:54.386495 5048 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.386546 5048 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:54.386575 5048 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.394462 5048 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.66:43517
I20240501 19:56:54.394541 5175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.66:43517 every 8 connection(s)
I20240501 19:56:54.395234 5048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:54.399019 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5048
I20240501 19:56:54.399127 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:54.401227 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.67:0
--local_ip_for_outbound_sockets=127.30.1.67
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:54.402995 5176 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:54.403173 5176 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:54.403542 5176 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:54.404332 4855 ts_manager.cc:194] Registered new tserver with Master: 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:56:54.404835 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.66:38789
W20240501 19:56:54.499982 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.500128 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.500186 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.503029 5179 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.503082 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.503175 5179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.67
I20240501 19:56:54.506451 5179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.67
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.507114 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.508023 5179 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:54.515509 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.516113 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.516849 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:54.517560 5179 server_base.cc:1034] running on GCE node
I20240501 19:56:54.517889 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:54.518347 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:54.519484 5179 hybrid_clock.cc:648] HybridClock initialized: now 1714593414519471 us; error 26 us; skew 500 ppm
I20240501 19:56:54.521052 5179 webserver.cc:460] Webserver started at http://127.30.1.67:34057/ using document root <none> and password file <none>
I20240501 19:56:54.521508 5179 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:54.521565 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:54.521723 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:54.523023 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4f272a2b03c244d69ff5595d457853ac"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.523427 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4f272a2b03c244d69ff5595d457853ac"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.526279 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:54.528126 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.529018 5179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:54.529114 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4f272a2b03c244d69ff5595d457853ac"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.529186 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:54.540544 5179 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:54.541047 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:54.541179 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.541592 5179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:54.542300 5179 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:54.542344 5179 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.542392 5179 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:54.542423 5179 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.550050 5179 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.67:46361
I20240501 19:56:54.550092 5306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.67:46361 every 8 connection(s)
I20240501 19:56:54.550792 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:54.556334 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5179
I20240501 19:56:54.556417 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:54.558147 5307 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:54.558320 5307 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:54.558490 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.68:0
--local_ip_for_outbound_sockets=127.30.1.68
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:54.558719 5307 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:54.559564 4855 ts_manager.cc:194] Registered new tserver with Master: 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:56:54.560077 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.67:48379
W20240501 19:56:54.654697 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.654875 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.654943 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.657874 5310 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.657935 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.658027 5310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.68
I20240501 19:56:54.661337 5310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.68
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.661954 5310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.662801 5310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:54.669876 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.670579 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.671221 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:54.672004 5310 server_base.cc:1034] running on GCE node
I20240501 19:56:54.672323 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:54.672791 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:54.673930 5310 hybrid_clock.cc:648] HybridClock initialized: now 1714593414673932 us; error 38 us; skew 500 ppm
I20240501 19:56:54.675392 5310 webserver.cc:460] Webserver started at http://127.30.1.68:46171/ using document root <none> and password file <none>
I20240501 19:56:54.675827 5310 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:54.675879 5310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:54.676038 5310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:54.677457 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6489e19b12b44216824eeb036c208ae5"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.677882 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6489e19b12b44216824eeb036c208ae5"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.680661 5310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:54.682593 5324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.683455 5310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:54.683543 5310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6489e19b12b44216824eeb036c208ae5"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.683615 5310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:54.707142 5310 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:54.707617 5310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:54.707754 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.708170 5310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:54.708895 5310 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:54.708971 5310 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.709018 5310 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:54.709053 5310 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.716662 5310 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.68:34679
I20240501 19:56:54.716702 5439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.68:34679 every 8 connection(s)
I20240501 19:56:54.717444 5310 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:54.723419 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5310
I20240501 19:56:54.723521 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:54.724784 5440 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:54.724977 5440 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:54.725435 5440 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:54.725507 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:0
--local_ip_for_outbound_sockets=127.30.1.69
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:54.726289 4855 ts_manager.cc:194] Registered new tserver with Master: 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679)
I20240501 19:56:54.726930 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.68:46623
W20240501 19:56:54.821727 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.821888 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.821954 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.824615 5443 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.824671 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.824751 5443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:56:54.828084 5443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.828738 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.829638 5443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:54.837041 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.837543 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:54.838292 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:54.838964 5443 server_base.cc:1034] running on GCE node
I20240501 19:56:54.839254 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:54.839741 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:54.840919 5443 hybrid_clock.cc:648] HybridClock initialized: now 1714593414840876 us; error 33 us; skew 500 ppm
I20240501 19:56:54.842311 5443 webserver.cc:460] Webserver started at http://127.30.1.69:34267/ using document root <none> and password file <none>
I20240501 19:56:54.842765 5443 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:54.842818 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:54.842978 5443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:54.844300 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4f5fc46abef54f898b8eccf16432f798"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.844704 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4f5fc46abef54f898b8eccf16432f798"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.847435 5443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:54.849364 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.850212 5443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:54.850308 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4f5fc46abef54f898b8eccf16432f798"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:56:54.850378 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:54.879598 5443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:54.880074 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:54.880214 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:54.880659 5443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:54.881462 5443 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:54.881510 5443 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.881563 5443 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:54.881594 5443 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:54.889412 5443 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:33827
I20240501 19:56:54.889444 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:33827 every 8 connection(s)
I20240501 19:56:54.890120 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:54.890461 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5443
I20240501 19:56:54.890579 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:54.892732 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:0
--local_ip_for_outbound_sockets=127.30.1.70
--webserver_interface=127.30.1.70
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:54.898085 5571 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:54.898258 5571 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:54.898602 5571 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:54.899430 4855 ts_manager.cc:194] Registered new tserver with Master: 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827)
I20240501 19:56:54.899895 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:56575
W20240501 19:56:54.988730 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:54.988895 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:54.989004 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:54.991662 5574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:54.991724 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:54.991806 5574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:56:54.995154 5574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:54.995884 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:54.996804 5574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:55.003851 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:55.004595 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:55.005213 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:55.005939 5574 server_base.cc:1034] running on GCE node
I20240501 19:56:55.006232 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:55.006649 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:55.007798 5574 hybrid_clock.cc:648] HybridClock initialized: now 1714593415007781 us; error 32 us; skew 500 ppm
I20240501 19:56:55.009359 5574 webserver.cc:460] Webserver started at http://127.30.1.70:41475/ using document root <none> and password file <none>
I20240501 19:56:55.009817 5574 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:55.009874 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:55.010026 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:55.011420 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "eaa8d49089fc428ab6e6601637abe221"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.011842 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "eaa8d49089fc428ab6e6601637abe221"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.014719 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:55.016593 5588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.017498 5574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:55.017601 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "eaa8d49089fc428ab6e6601637abe221"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.017676 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:55.043413 5574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:55.043902 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:55.044036 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:55.044466 5574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:55.045202 5574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:55.045248 5574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.045307 5574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:55.045334 5574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.053309 5574 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:41385
I20240501 19:56:55.053325 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:41385 every 8 connection(s)
I20240501 19:56:55.054037 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:55.057515 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5574
I20240501 19:56:55.057643 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:55.059559 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:0
--local_ip_for_outbound_sockets=127.30.1.71
--webserver_interface=127.30.1.71
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:56:55.061762 5702 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:55.061954 5702 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:55.062387 5702 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:55.063314 4855 ts_manager.cc:194] Registered new tserver with Master: eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:56:55.064190 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:40705
W20240501 19:56:55.155771 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:55.155918 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:55.155956 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:55.158668 5705 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:56:55.158718 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:55.158779 5705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:56:55.161840 5705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:55.162469 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:55.163292 5705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:55.170364 5711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:55.171033 5713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:55.171659 5710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:55.172425 5705 server_base.cc:1034] running on GCE node
I20240501 19:56:55.172734 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:55.173224 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:55.174376 5705 hybrid_clock.cc:648] HybridClock initialized: now 1714593415174364 us; error 28 us; skew 500 ppm
I20240501 19:56:55.175927 5705 webserver.cc:460] Webserver started at http://127.30.1.71:41261/ using document root <none> and password file <none>
I20240501 19:56:55.176383 5705 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:55.176436 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:55.176586 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:55.178014 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4b2c3262271245a4b1acf221c8e5110f"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.178429 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4b2c3262271245a4b1acf221c8e5110f"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.181262 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:55.183197 5719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.184055 5705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:55.184149 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4b2c3262271245a4b1acf221c8e5110f"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:56:55.184222 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:55.193987 5705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:55.194469 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:55.194612 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:55.195034 5705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:55.195749 5705 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:55.195796 5705 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.195847 5705 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:55.195879 5705 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:55.203644 5705 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:35049
I20240501 19:56:55.203693 5832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:35049 every 8 connection(s)
I20240501 19:56:55.204385 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:55.204564 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 5705
I20240501 19:56:55.204677 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:55.211910 5833 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:56:55.212064 5833 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:55.212384 5833 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:56:55.213135 4855 ts_manager.cc:194] Registered new tserver with Master: 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049)
I20240501 19:56:55.213601 4855 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:46093
I20240501 19:56:55.218269 30725 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:55.225960 4855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:56:55.240741 5241 tablet_service.cc:1467] Processing CreateTablet for tablet a054a3f0d0ca40d98979620a3bf39a1d (DEFAULT_TABLE table=TestTable [id=c74331f516e2486c896a4f5b5b485c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.241631 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a054a3f0d0ca40d98979620a3bf39a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.245877 5849 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:56:55.246783 5636 tablet_service.cc:1467] Processing CreateTablet for tablet a054a3f0d0ca40d98979620a3bf39a1d (DEFAULT_TABLE table=TestTable [id=c74331f516e2486c896a4f5b5b485c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:55.247458 5636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a054a3f0d0ca40d98979620a3bf39a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.247678 5849 tablet_bootstrap.cc:654] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.248378 5849 log.cc:824] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:55.249073 5045 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:56:55.249835 5849 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:56:55.249989 5849 ts_tablet_manager.cc:1367] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:55.252045 5851 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:56:55.252393 5849 raft_consensus.cc:348] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.252584 5849 raft_consensus.cc:374] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.252641 5849 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.253077 5849 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.253629 5307 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:56:55.253893 5849 ts_tablet_manager.cc:1398] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:55.253986 5851 tablet_bootstrap.cc:654] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:55.254583 5851 log.cc:824] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:55.255745 5851 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: No bootstrap required, opened a new log
I20240501 19:56:55.255884 5851 ts_tablet_manager.cc:1367] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:55.258610 5851 raft_consensus.cc:348] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.258749 5851 raft_consensus.cc:374] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:55.258798 5851 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:56:55.259097 5851 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.259584 5702 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:56:55.259687 5851 ts_tablet_manager.cc:1398] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20240501 19:56:55.301985 5308 tablet.cc:2369] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:55.304724 5703 tablet.cc:2369] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:55.406378 5176 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:56:55.656329 5852 raft_consensus.cc:482] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:55.656507 5852 raft_consensus.cc:504] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.657433 5852 leader_election.cc:288] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:56:55.660892 5656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa8d49089fc428ab6e6601637abe221" is_pre_election: true
I20240501 19:56:55.661152 5656 raft_consensus.cc:2454] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:56:55.661579 5195 leader_election.cc:302] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, eaa8d49089fc428ab6e6601637abe221; no voters:
I20240501 19:56:55.661828 5852 raft_consensus.cc:2781] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:55.661895 5852 raft_consensus.cc:482] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:55.661947 5852 raft_consensus.cc:3037] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.662838 5852 raft_consensus.cc:504] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.663069 5852 leader_election.cc:288] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:56:55.663403 5656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa8d49089fc428ab6e6601637abe221"
I20240501 19:56:55.663519 5656 raft_consensus.cc:3037] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:55.664305 5656 raft_consensus.cc:2454] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:56:55.664558 5195 leader_election.cc:302] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, eaa8d49089fc428ab6e6601637abe221; no voters:
I20240501 19:56:55.664721 5852 raft_consensus.cc:2781] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:55.664947 5852 raft_consensus.cc:686] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:56:55.665223 5852 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:56:55.667603 4855 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:56:55.686151 30725 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:55.687391 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b051b1f42cb443e58813b92c5ae4a62e to finish bootstrapping
I20240501 19:56:55.692701 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 43061d2e1908476eaf2b3bdc9a402ee2 to finish bootstrapping
I20240501 19:56:55.697468 30725 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4f272a2b03c244d69ff5595d457853ac to finish bootstrapping
I20240501 19:56:55.701373 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6489e19b12b44216824eeb036c208ae5 to finish bootstrapping
I20240501 19:56:55.705879 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4f5fc46abef54f898b8eccf16432f798 to finish bootstrapping
I20240501 19:56:55.710328 30725 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eaa8d49089fc428ab6e6601637abe221 to finish bootstrapping
I20240501 19:56:55.714079 30725 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4b2c3262271245a4b1acf221c8e5110f to finish bootstrapping
I20240501 19:56:55.719031 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5310
I20240501 19:56:55.726706 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5443
I20240501 19:56:55.732057 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5574
I20240501 19:56:55.737676 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5705
W20240501 19:56:56.268460 5195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.70:41385: connect: Connection refused (error 111)
W20240501 19:56:56.269260 5195 consensus_peers.cc:485] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.70:41385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:58.598439 5195 consensus_peers.cc:485] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.70:41385: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:56:59.493556 4855 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:59.505573 4979 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2013d6f6b496e99af4a248bd3ed57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.506515 4979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2013d6f6b496e99af4a248bd3ed57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.507220 5110 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca2013d6f6b496e99af4a248bd3ed57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.507828 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2013d6f6b496e99af4a248bd3ed57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.509171 5109 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbdfb0387a9449f897a454ff26276b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.509337 5108 tablet_service.cc:1467] Processing CreateTablet for tablet a3fd829e098745598ccb473e5a0503b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.509454 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbdfb0387a9449f897a454ff26276b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.509577 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd829e098745598ccb473e5a0503b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.505573 4977 tablet_service.cc:1467] Processing CreateTablet for tablet a3fd829e098745598ccb473e5a0503b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.505573 4978 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbdfb0387a9449f897a454ff26276b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:59.511312 4977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd829e098745598ccb473e5a0503b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.511451 4978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbdfb0387a9449f897a454ff26276b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:59.513316 5886 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:56:59.514341 5885 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:56:59.515419 5886 tablet_bootstrap.cc:654] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.516140 5885 tablet_bootstrap.cc:654] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.516451 5886 log.cc:824] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:59.517099 5885 log.cc:824] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:59.518033 5886 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:56:59.518178 5886 ts_tablet_manager.cc:1367] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:59.518584 5885 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:56:59.518735 5885 ts_tablet_manager.cc:1367] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:59.520658 5886 raft_consensus.cc:348] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.520802 5886 raft_consensus.cc:374] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.520843 5886 raft_consensus.cc:729] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.521122 5885 raft_consensus.cc:348] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.521196 5886 consensus_queue.cc:260] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.521266 5885 raft_consensus.cc:374] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.521337 5885 raft_consensus.cc:729] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.521680 5885 consensus_queue.cc:260] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.521883 5886 ts_tablet_manager.cc:1398] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:59.522189 5886 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:56:59.522403 5885 ts_tablet_manager.cc:1398] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:59.522614 5885 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:56:59.522950 5886 tablet_bootstrap.cc:654] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.523442 5885 tablet_bootstrap.cc:654] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.524127 5886 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:56:59.524197 5886 ts_tablet_manager.cc:1367] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:59.524698 5886 raft_consensus.cc:348] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.524803 5886 raft_consensus.cc:374] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.524847 5886 raft_consensus.cc:729] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.524991 5886 consensus_queue.cc:260] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.525251 5886 ts_tablet_manager.cc:1398] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.525337 5885 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:56:59.525395 5886 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:56:59.525400 5885 ts_tablet_manager.cc:1367] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:59.525871 5885 raft_consensus.cc:348] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.525952 5885 raft_consensus.cc:374] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.525985 5885 raft_consensus.cc:729] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.526165 5886 tablet_bootstrap.cc:654] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.526109 5885 consensus_queue.cc:260] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.526437 5885 ts_tablet_manager.cc:1398] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:59.526589 5885 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:56:59.527210 5886 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:56:59.527281 5886 ts_tablet_manager.cc:1367] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:59.527482 5885 tablet_bootstrap.cc:654] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:59.527720 5886 raft_consensus.cc:348] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.527806 5886 raft_consensus.cc:374] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.527837 5886 raft_consensus.cc:729] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.527943 5886 consensus_queue.cc:260] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.528246 5886 ts_tablet_manager.cc:1398] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.528560 5885 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:56:59.528627 5885 ts_tablet_manager.cc:1367] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:59.529027 5885 raft_consensus.cc:348] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.529119 5885 raft_consensus.cc:374] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:59.529168 5885 raft_consensus.cc:729] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:56:59.529278 5885 consensus_queue.cc:260] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.529537 5885 ts_tablet_manager.cc:1398] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:59.582094 5890 raft_consensus.cc:482] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.582271 5890 raft_consensus.cc:504] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.583014 5890 leader_election.cc:288] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:56:59.586552 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2013d6f6b496e99af4a248bd3ed57" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:56:59.586783 5130 raft_consensus.cc:2454] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:56:59.587154 4933 leader_election.cc:302] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.587366 5890 raft_consensus.cc:2781] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.587426 5890 raft_consensus.cc:482] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.587455 5890 raft_consensus.cc:3037] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.588213 5890 raft_consensus.cc:504] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.588447 5890 leader_election.cc:288] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:56:59.588758 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca2013d6f6b496e99af4a248bd3ed57" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:56:59.588840 5130 raft_consensus.cc:3037] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.589603 5130 raft_consensus.cc:2454] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:56:59.589856 4933 leader_election.cc:302] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.590010 5890 raft_consensus.cc:2781] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.590178 5890 raft_consensus.cc:686] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:56:59.590437 5890 consensus_queue.cc:237] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:56:59.592519 4856 catalog_manager.cc:5484] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } }
W20240501 19:56:59.649091 5177 tablet.cc:2369] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:59.721401 5889 raft_consensus.cc:482] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.721505 5889 raft_consensus.cc:504] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.721575 5890 raft_consensus.cc:482] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.721657 5890 raft_consensus.cc:504] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.721923 5890 leader_election.cc:288] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:56:59.722322 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:56:59.722440 5130 raft_consensus.cc:2454] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:56:59.722460 5889 leader_election.cc:288] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:56:59.723024 4933 leader_election.cc:302] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.723237 5890 raft_consensus.cc:2781] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.723292 5890 raft_consensus.cc:482] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.723346 5890 raft_consensus.cc:3037] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.724076 5890 raft_consensus.cc:504] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.724323 5890 leader_election.cc:288] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:56:59.724635 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:56:59.724733 5130 raft_consensus.cc:3037] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.725615 5130 raft_consensus.cc:2454] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:56:59.725862 4933 leader_election.cc:302] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.725929 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e" is_pre_election: true
I20240501 19:56:59.726027 5890 raft_consensus.cc:2781] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.726116 5890 raft_consensus.cc:686] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:56:59.726266 5890 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.727294 5066 leader_election.cc:302] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2; no voters: b051b1f42cb443e58813b92c5ae4a62e
I20240501 19:56:59.727490 5889 raft_consensus.cc:2726] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:59.728130 4856 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
W20240501 19:56:59.740145 5046 tablet.cc:2369] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:59.857733 5889 raft_consensus.cc:482] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:59.857839 5889 raft_consensus.cc:504] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.858141 5889 leader_election.cc:288] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:56:59.858526 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbdfb0387a9449f897a454ff26276b4" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e" is_pre_election: true
I20240501 19:56:59.858673 4999 raft_consensus.cc:2454] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 0.
I20240501 19:56:59.858955 5066 leader_election.cc:302] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.859103 5889 raft_consensus.cc:2781] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:59.859166 5889 raft_consensus.cc:482] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:59.859200 5889 raft_consensus.cc:3037] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.859972 5889 raft_consensus.cc:504] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.860178 5889 leader_election.cc:288] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:56:59.860486 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbdfb0387a9449f897a454ff26276b4" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:56:59.860592 4999 raft_consensus.cc:3037] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:59.861361 4999 raft_consensus.cc:2454] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 1.
I20240501 19:56:59.861617 5066 leader_election.cc:302] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:56:59.861757 5889 raft_consensus.cc:2781] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:59.861981 5889 raft_consensus.cc:686] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:56:59.862268 5889 consensus_queue.cc:237] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:56:59.864490 4856 catalog_manager.cc:5484] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
W20240501 19:57:00.044603 5905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:00.044802 5905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:00.047731 5905 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:57:00.111896 5890 consensus_queue.cc:1035] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.136709 4999 raft_consensus.cc:1261] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.137229 5889 consensus_queue.cc:1035] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.139227 5129 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.139642 5896 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.149603 5930 mvcc.cc:205] Tried to move back new op lower bound from 7022974648881139712 to 7022974647199498240. Current Snapshot: MvccSnapshot[applied={T|T < 7022974648881139712}]
I20240501 19:57:00.152460 5938 mvcc.cc:205] Tried to move back new op lower bound from 7022974648919760896 to 7022974647199498240. Current Snapshot: MvccSnapshot[applied={T|T < 7022974648881139712}]
INSERT report
rows total: 2000
time total: 91.3986 ms
time per row: 0.0456993 ms
I20240501 19:57:00.178267 4856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:00.183008 4977 tablet_service.cc:1467] Processing CreateTablet for tablet d1656d575d7f4d79bb289c3a043cf6aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.183275 4977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1656d575d7f4d79bb289c3a043cf6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.183368 5110 tablet_service.cc:1467] Processing CreateTablet for tablet cfb7ca52d5ca4789888fe71d8b29b65f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.183595 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb7ca52d5ca4789888fe71d8b29b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.183749 5240 tablet_service.cc:1467] Processing CreateTablet for tablet cfb7ca52d5ca4789888fe71d8b29b65f (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.184046 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb7ca52d5ca4789888fe71d8b29b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.184355 4978 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb072bde014a35989544648d90249b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.184576 4978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb072bde014a35989544648d90249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.185564 5241 tablet_service.cc:1467] Processing CreateTablet for tablet d1656d575d7f4d79bb289c3a043cf6aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.185798 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1656d575d7f4d79bb289c3a043cf6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.185837 5954 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:00.186610 5240 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb072bde014a35989544648d90249b (DEFAULT_TABLE table=rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.186663 5954 tablet_bootstrap.cc:654] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.186889 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb072bde014a35989544648d90249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.187443 5955 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:00.188149 5954 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:00.188202 5954 ts_tablet_manager.cc:1367] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.188477 5955 tablet_bootstrap.cc:654] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.188508 5954 raft_consensus.cc:348] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.188570 5954 raft_consensus.cc:374] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.188591 5954 raft_consensus.cc:729] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.188726 5954 consensus_queue.cc:260] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.188748 5956 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:00.189323 5954 ts_tablet_manager.cc:1398] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.189615 5956 tablet_bootstrap.cc:654] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.189615 5954 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:00.190418 5954 tablet_bootstrap.cc:654] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.190791 5955 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:00.190871 5955 ts_tablet_manager.cc:1367] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:00.190974 5956 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:00.191035 5956 ts_tablet_manager.cc:1367] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.191280 5955 raft_consensus.cc:348] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.191363 5955 raft_consensus.cc:374] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.191391 5955 raft_consensus.cc:729] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.191490 5955 consensus_queue.cc:260] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.191560 5956 raft_consensus.cc:348] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.191650 5956 raft_consensus.cc:374] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.191682 5956 raft_consensus.cc:729] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.191769 5955 ts_tablet_manager.cc:1398] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.191789 5956 consensus_queue.cc:260] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.192034 5956 ts_tablet_manager.cc:1398] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:00.192353 5956 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:00.192502 5954 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:00.192981 5954 ts_tablet_manager.cc:1367] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:00.193238 5956 tablet_bootstrap.cc:654] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.194023 5954 raft_consensus.cc:348] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.194171 5954 raft_consensus.cc:374] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.194255 5954 raft_consensus.cc:729] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.194413 5954 consensus_queue.cc:260] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.194769 5956 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:00.194847 5956 ts_tablet_manager.cc:1367] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:00.195005 5954 ts_tablet_manager.cc:1398] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:00.195227 5956 raft_consensus.cc:348] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.195655 5956 raft_consensus.cc:374] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.195705 5956 raft_consensus.cc:729] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.195804 5956 consensus_queue.cc:260] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.196059 5956 ts_tablet_manager.cc:1398] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.196218 5956 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:00.197610 5956 tablet_bootstrap.cc:654] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.198580 5956 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:00.198643 5956 ts_tablet_manager.cc:1367] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.198963 5956 raft_consensus.cc:348] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.199036 5956 raft_consensus.cc:374] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.199059 5956 raft_consensus.cc:729] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.199182 5956 consensus_queue.cc:260] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.199394 5956 ts_tablet_manager.cc:1398] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:00.418589 5890 raft_consensus.cc:482] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.418690 5890 raft_consensus.cc:504] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.419014 5890 leader_election.cc:288] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:00.422458 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1656d575d7f4d79bb289c3a043cf6aa" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac" is_pre_election: true
I20240501 19:57:00.422649 5261 raft_consensus.cc:2454] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:57:00.422971 4934 leader_election.cc:302] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:00.423148 5890 raft_consensus.cc:2781] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.423202 5890 raft_consensus.cc:482] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.423246 5890 raft_consensus.cc:3037] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.423961 5890 raft_consensus.cc:504] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.424213 5890 leader_election.cc:288] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:00.424502 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1656d575d7f4d79bb289c3a043cf6aa" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:00.424597 5261 raft_consensus.cc:3037] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.425400 5261 raft_consensus.cc:2454] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:57:00.425652 4934 leader_election.cc:302] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:00.425791 5890 raft_consensus.cc:2781] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.425882 5890 raft_consensus.cc:686] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:00.426005 5890 consensus_queue.cc:237] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.427260 4856 catalog_manager.cc:5484] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } }
I20240501 19:57:00.439065 5951 raft_consensus.cc:482] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.439148 5951 raft_consensus.cc:504] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.439457 5951 leader_election.cc:288] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:00.442260 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:57:00.442366 5130 raft_consensus.cc:2454] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:57:00.442677 5196 leader_election.cc:302] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:00.442843 5951 raft_consensus.cc:2781] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.442894 5951 raft_consensus.cc:482] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.442921 5951 raft_consensus.cc:3037] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.443571 5951 raft_consensus.cc:504] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.443802 5951 leader_election.cc:288] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:00.444144 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:00.444279 5130 raft_consensus.cc:3037] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.444916 5130 raft_consensus.cc:2454] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:57:00.445194 5196 leader_election.cc:302] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:00.445376 5951 raft_consensus.cc:2781] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.445559 5951 raft_consensus.cc:686] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:00.445675 5951 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.446893 4856 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
I20240501 19:57:00.610313 5951 raft_consensus.cc:482] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:00.610414 5951 raft_consensus.cc:504] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.610704 5951 leader_election.cc:288] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:00.613790 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb072bde014a35989544648d90249b" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e" is_pre_election: true
I20240501 19:57:00.613915 4999 raft_consensus.cc:2454] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:57:00.614130 5197 leader_election.cc:302] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:00.614287 5951 raft_consensus.cc:2781] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:00.614341 5951 raft_consensus.cc:482] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:00.614367 5951 raft_consensus.cc:3037] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.615013 5951 raft_consensus.cc:504] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.615265 5951 leader_election.cc:288] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:00.615566 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb072bde014a35989544648d90249b" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:00.615649 4999 raft_consensus.cc:3037] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:00.616318 4999 raft_consensus.cc:2454] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:57:00.616571 5197 leader_election.cc:302] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:00.616709 5951 raft_consensus.cc:2781] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:00.616788 5951 raft_consensus.cc:686] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:00.616941 5951 consensus_queue.cc:237] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } }
I20240501 19:57:00.618345 4856 catalog_manager.cc:5484] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } }
W20240501 19:57:00.728708 5970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:00.728924 5970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:00.731842 5970 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:57:00.816386 5130 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.816386 4999 raft_consensus.cc:1261] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.816828 5951 consensus_queue.cc:1035] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.817189 5951 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.817967 5261 raft_consensus.cc:1261] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:00.819401 5890 consensus_queue.cc:1035] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:00.824381 5999 mvcc.cc:205] Tried to move back new op lower bound from 7022974651657842688 to 7022974650146250752. Current Snapshot: MvccSnapshot[applied={T|T < 7022974651657842688}]
INSERT report
rows total: 2000
time total: 91.5278 ms
time per row: 0.0457639 ms
I20240501 19:57:00.860375 4848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:00.865098 5110 tablet_service.cc:1467] Processing CreateTablet for tablet a7191b815b8747a082c26c1099a65518 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.865307 5240 tablet_service.cc:1467] Processing CreateTablet for tablet a7191b815b8747a082c26c1099a65518 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.865389 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7191b815b8747a082c26c1099a65518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.865409 5109 tablet_service.cc:1467] Processing CreateTablet for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.865568 5240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7191b815b8747a082c26c1099a65518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.865615 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.865828 5241 tablet_service.cc:1467] Processing CreateTablet for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.866024 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.867591 5108 tablet_service.cc:1467] Processing CreateTablet for tablet aeaad9f75c694c84a890194769b4b461 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.867794 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaad9f75c694c84a890194769b4b461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.868062 6019 tablet_bootstrap.cc:492] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:00.868115 4978 tablet_service.cc:1467] Processing CreateTablet for tablet aeaad9f75c694c84a890194769b4b461 (DEFAULT_TABLE table=rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:00.868352 4978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaad9f75c694c84a890194769b4b461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:00.868779 6019 tablet_bootstrap.cc:654] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.869729 6019 tablet_bootstrap.cc:492] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:00.869805 6019 ts_tablet_manager.cc:1367] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:00.870079 6020 tablet_bootstrap.cc:492] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:00.870148 6019 raft_consensus.cc:348] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.870246 6019 raft_consensus.cc:374] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.870321 6019 raft_consensus.cc:729] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.870450 6019 consensus_queue.cc:260] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.870736 6019 ts_tablet_manager.cc:1398] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:00.870924 6019 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:00.870920 6022 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:00.871073 6020 tablet_bootstrap.cc:654] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.871763 6019 tablet_bootstrap.cc:654] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.871763 6022 tablet_bootstrap.cc:654] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.872277 6020 tablet_bootstrap.cc:492] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:00.872382 6020 ts_tablet_manager.cc:1367] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:00.872792 6020 raft_consensus.cc:348] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.872892 6020 raft_consensus.cc:374] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.872944 6020 raft_consensus.cc:729] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.873096 6019 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:00.873055 6020 consensus_queue.cc:260] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:00.873160 6019 ts_tablet_manager.cc:1367] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:00.873322 6020 ts_tablet_manager.cc:1398] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:00.873414 6022 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:00.873461 6020 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:00.873483 6022 ts_tablet_manager.cc:1367] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:00.873548 6019 raft_consensus.cc:348] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.873621 6019 raft_consensus.cc:374] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.873661 6019 raft_consensus.cc:729] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.873760 6019 consensus_queue.cc:260] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.873884 6022 raft_consensus.cc:348] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.873971 6022 raft_consensus.cc:374] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.874001 6022 raft_consensus.cc:729] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.874099 6022 consensus_queue.cc:260] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.874181 6020 tablet_bootstrap.cc:654] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.874433 6022 ts_tablet_manager.cc:1398] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:00.875056 6019 ts_tablet_manager.cc:1398] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.875368 6020 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:00.875577 6020 ts_tablet_manager.cc:1367] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:00.876008 6020 raft_consensus.cc:348] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.876106 6020 raft_consensus.cc:374] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.876138 6020 raft_consensus.cc:729] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.876327 6020 consensus_queue.cc:260] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.876650 6020 ts_tablet_manager.cc:1398] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:00.876806 6020 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:00.877599 6020 tablet_bootstrap.cc:654] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:00.878566 6020 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:00.878648 6020 ts_tablet_manager.cc:1367] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:00.879066 6020 raft_consensus.cc:348] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.879174 6020 raft_consensus.cc:374] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:00.879244 6020 raft_consensus.cc:729] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:00.879387 6020 consensus_queue.cc:260] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:00.879631 6020 ts_tablet_manager.cc:1398] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.040303 5889 raft_consensus.cc:482] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.040434 5889 raft_consensus.cc:504] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.040834 5889 leader_election.cc:288] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.044309 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac" is_pre_election: true
I20240501 19:57:01.044449 5261 raft_consensus.cc:2454] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 0.
I20240501 19:57:01.044708 5066 leader_election.cc:302] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.044871 5889 raft_consensus.cc:2781] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.044941 5889 raft_consensus.cc:482] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.044968 5889 raft_consensus.cc:3037] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.045676 5889 raft_consensus.cc:504] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.045902 5889 leader_election.cc:288] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.046226 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:01.046319 5261 raft_consensus.cc:3037] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.047096 5261 raft_consensus.cc:2454] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 1.
I20240501 19:57:01.047351 5066 leader_election.cc:302] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.047508 5889 raft_consensus.cc:2781] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.047688 5889 raft_consensus.cc:686] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:57:01.047842 5889 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.049270 4848 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
I20240501 19:57:01.051383 5896 raft_consensus.cc:482] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.051472 5896 raft_consensus.cc:504] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.051769 5896 leader_election.cc:288] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.052083 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaad9f75c694c84a890194769b4b461" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:57:01.052201 5130 raft_consensus.cc:2454] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:57:01.052486 4933 leader_election.cc:302] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.052667 5896 raft_consensus.cc:2781] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.052722 5896 raft_consensus.cc:482] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.052753 5896 raft_consensus.cc:3037] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.053478 5896 raft_consensus.cc:504] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.053714 5896 leader_election.cc:288] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.054011 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaad9f75c694c84a890194769b4b461" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:01.054111 5130 raft_consensus.cc:3037] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.054842 5130 raft_consensus.cc:2454] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:57:01.055073 4933 leader_election.cc:302] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.055212 5896 raft_consensus.cc:2781] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.055294 5896 raft_consensus.cc:686] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:01.055414 5896 consensus_queue.cc:237] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.056564 4848 catalog_manager.cc:5484] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
I20240501 19:57:01.075711 5889 raft_consensus.cc:482] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.075795 5889 raft_consensus.cc:504] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.076035 5889 leader_election.cc:288] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.076385 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7191b815b8747a082c26c1099a65518" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac" is_pre_election: true
I20240501 19:57:01.076494 5261 raft_consensus.cc:2454] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 0.
I20240501 19:57:01.076733 5066 leader_election.cc:302] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.076897 5889 raft_consensus.cc:2781] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.076972 5889 raft_consensus.cc:482] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.077029 5889 raft_consensus.cc:3037] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.077668 5889 raft_consensus.cc:504] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.077890 5889 leader_election.cc:288] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.078195 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7191b815b8747a082c26c1099a65518" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:01.078290 5261 raft_consensus.cc:3037] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.078965 5261 raft_consensus.cc:2454] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 1.
I20240501 19:57:01.079216 5066 leader_election.cc:302] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.079396 5889 raft_consensus.cc:2781] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.079520 5889 raft_consensus.cc:686] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:57:01.079655 5889 consensus_queue.cc:237] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.081040 4848 catalog_manager.cc:5484] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } }
W20240501 19:57:01.188659 5195 consensus_peers.cc:485] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.70:41385: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.197876 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:01.198123 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:01.201215 6033 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:57:01.282600 5261 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.283016 5889 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.284497 5260 raft_consensus.cc:1261] T a7191b815b8747a082c26c1099a65518 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.286094 5889 consensus_queue.cc:1035] T a7191b815b8747a082c26c1099a65518 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.289252 5130 raft_consensus.cc:1261] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.289673 5896 consensus_queue.cc:1035] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.5187 ms
time per row: 0.0442594 ms
I20240501 19:57:01.326429 4856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:01.331104 5241 tablet_service.cc:1467] Processing CreateTablet for tablet 259f0c2339954fa7b9a1d4e604a8511a (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.331108 4977 tablet_service.cc:1467] Processing CreateTablet for tablet fe3059729d0d4d7ebf4c2f39071919f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.331104 4978 tablet_service.cc:1467] Processing CreateTablet for tablet 259f0c2339954fa7b9a1d4e604a8511a (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.331400 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259f0c2339954fa7b9a1d4e604a8511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.331422 4977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3059729d0d4d7ebf4c2f39071919f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.331775 5109 tablet_service.cc:1467] Processing CreateTablet for tablet e3613e90b6a747cbb4e1bf8eb9107675 (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.332003 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3613e90b6a747cbb4e1bf8eb9107675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.333113 6019 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:01.333520 6022 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:01.333829 6019 tablet_bootstrap.cc:654] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.334010 6020 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:01.334306 6022 tablet_bootstrap.cc:654] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.334761 6020 tablet_bootstrap.cc:654] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.334911 4978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259f0c2339954fa7b9a1d4e604a8511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.334919 6019 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:01.334988 6019 ts_tablet_manager.cc:1367] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.335387 6019 raft_consensus.cc:348] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.335474 6019 raft_consensus.cc:374] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.335510 6019 raft_consensus.cc:729] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.335613 6019 consensus_queue.cc:260] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.335693 4977 tablet_service.cc:1467] Processing CreateTablet for tablet e3613e90b6a747cbb4e1bf8eb9107675 (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.335814 5241 tablet_service.cc:1467] Processing CreateTablet for tablet fe3059729d0d4d7ebf4c2f39071919f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.335873 4977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3613e90b6a747cbb4e1bf8eb9107675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.335871 6019 ts_tablet_manager.cc:1398] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.336117 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3059729d0d4d7ebf4c2f39071919f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.337188 6020 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:01.337188 6022 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:01.337256 6020 ts_tablet_manager.cc:1367] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:01.337256 6022 ts_tablet_manager.cc:1367] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:01.337641 6022 raft_consensus.cc:348] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.337639 6020 raft_consensus.cc:348] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.337725 6020 raft_consensus.cc:374] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.337725 6022 raft_consensus.cc:374] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.337760 6022 raft_consensus.cc:729] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.337759 6020 raft_consensus.cc:729] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.337877 6022 consensus_queue.cc:260] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.337877 6020 consensus_queue.cc:260] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.338109 6020 ts_tablet_manager.cc:1398] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.338119 6022 ts_tablet_manager.cc:1398] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:01.338260 6022 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:01.339175 6022 tablet_bootstrap.cc:654] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.340272 6022 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:01.340293 6019 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:01.340343 6022 ts_tablet_manager.cc:1367] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:01.340682 6022 raft_consensus.cc:348] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.340773 6022 raft_consensus.cc:374] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.340808 6022 raft_consensus.cc:729] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.340968 6022 consensus_queue.cc:260] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.341076 6019 tablet_bootstrap.cc:654] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.341249 6022 ts_tablet_manager.cc:1398] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:01.341409 6022 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:01.342278 6019 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:01.342362 6019 ts_tablet_manager.cc:1367] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.342670 6019 raft_consensus.cc:348] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.342752 6019 raft_consensus.cc:374] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.342800 6019 raft_consensus.cc:729] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.342909 6022 tablet_bootstrap.cc:654] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.342919 6019 consensus_queue.cc:260] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.343155 6019 ts_tablet_manager.cc:1398] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.343912 6022 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:01.343982 6022 ts_tablet_manager.cc:1367] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:01.344277 6022 raft_consensus.cc:348] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.344357 6022 raft_consensus.cc:374] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.344388 6022 raft_consensus.cc:729] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.344516 6022 consensus_queue.cc:260] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.344731 6022 ts_tablet_manager.cc:1398] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.401381 5896 raft_consensus.cc:482] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.401453 5896 raft_consensus.cc:504] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.401692 5896 leader_election.cc:288] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.401993 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:57:01.402101 5130 raft_consensus.cc:2454] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:57:01.402375 4933 leader_election.cc:302] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.402530 5896 raft_consensus.cc:2781] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.402591 5896 raft_consensus.cc:482] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.402642 5896 raft_consensus.cc:3037] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.403487 5896 raft_consensus.cc:504] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.403739 5896 leader_election.cc:288] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.404017 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:01.404106 5130 raft_consensus.cc:3037] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.404834 5130 raft_consensus.cc:2454] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:57:01.405108 4933 leader_election.cc:302] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.405292 5896 raft_consensus.cc:2781] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.405378 5896 raft_consensus.cc:686] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:01.405483 5896 consensus_queue.cc:237] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.406710 4856 catalog_manager.cc:5484] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
I20240501 19:57:01.468578 6014 raft_consensus.cc:482] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.468663 6014 raft_consensus.cc:504] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.468933 6014 leader_election.cc:288] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:01.469286 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259f0c2339954fa7b9a1d4e604a8511a" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e" is_pre_election: true
I20240501 19:57:01.469388 4999 raft_consensus.cc:2454] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:57:01.469627 5197 leader_election.cc:302] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.469781 6014 raft_consensus.cc:2781] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.469836 6014 raft_consensus.cc:482] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.469864 6014 raft_consensus.cc:3037] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.470522 6014 raft_consensus.cc:504] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.470734 6014 leader_election.cc:288] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:01.471029 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259f0c2339954fa7b9a1d4e604a8511a" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:01.471112 4999 raft_consensus.cc:3037] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.471678 4999 raft_consensus.cc:2454] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:57:01.471923 5197 leader_election.cc:302] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.472105 6014 raft_consensus.cc:2781] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.472293 6014 raft_consensus.cc:686] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:01.472420 6014 consensus_queue.cc:237] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.473644 4856 catalog_manager.cc:5484] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } }
W20240501 19:57:01.613941 5195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.70:41385: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 19:57:01.675879 5896 raft_consensus.cc:482] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.675980 5896 raft_consensus.cc:504] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.676280 5896 leader_election.cc:288] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.676667 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac" is_pre_election: true
I20240501 19:57:01.676792 5261 raft_consensus.cc:2454] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 0.
I20240501 19:57:01.677099 4934 leader_election.cc:302] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.677275 5896 raft_consensus.cc:2781] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.677330 5896 raft_consensus.cc:482] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.677361 5896 raft_consensus.cc:3037] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.678087 5896 raft_consensus.cc:504] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.678320 5896 leader_election.cc:288] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:01.678622 5261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6" candidate_uuid: "b051b1f42cb443e58813b92c5ae4a62e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:01.678709 5261 raft_consensus.cc:3037] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.679412 5261 raft_consensus.cc:2454] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b051b1f42cb443e58813b92c5ae4a62e in term 1.
I20240501 19:57:01.679665 4934 leader_election.cc:302] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:01.679822 5896 raft_consensus.cc:2781] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.679896 5896 raft_consensus.cc:686] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Leader. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:01.680018 5896 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.681422 4856 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: term changed from 0 to 1, leader changed from <none> to b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65). New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } }
W20240501 19:57:01.790640 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:01.790843 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:01.793836 6082 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:57:01.813151 5896 consensus_queue.cc:1035] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.863799 5261 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.864280 5896 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:01.866034 4999 raft_consensus.cc:1261] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:01.866441 6014 consensus_queue.cc:1035] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 73.0252 ms
time per row: 0.0365126 ms
I20240501 19:57:01.905000 4856 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35964:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:01.909817 5241 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0753c139624897b37cab9f3a9eab35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.909909 5109 tablet_service.cc:1467] Processing CreateTablet for tablet 9b0753c139624897b37cab9f3a9eab35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.910104 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0753c139624897b37cab9f3a9eab35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.910238 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0753c139624897b37cab9f3a9eab35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.911810 6133 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:01.911898 6134 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:01.911944 5108 tablet_service.cc:1467] Processing CreateTablet for tablet fc16a5fd751d4f109b420ed953abe053 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.912179 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc16a5fd751d4f109b420ed953abe053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.912719 6133 tablet_bootstrap.cc:654] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.912809 6134 tablet_bootstrap.cc:654] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.913095 5109 tablet_service.cc:1467] Processing CreateTablet for tablet ecbb6869820148cbb31738a0efc7c515 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.913304 5109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecbb6869820148cbb31738a0efc7c515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.913235 4977 tablet_service.cc:1467] Processing CreateTablet for tablet ecbb6869820148cbb31738a0efc7c515 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.913558 4977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecbb6869820148cbb31738a0efc7c515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.914264 6133 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:01.914327 6133 ts_tablet_manager.cc:1367] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:01.914705 6133 raft_consensus.cc:348] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.914796 6133 raft_consensus.cc:374] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.914822 6133 raft_consensus.cc:729] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.914952 6133 consensus_queue.cc:260] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.915212 6133 ts_tablet_manager.cc:1398] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:01.915414 6137 tablet_bootstrap.cc:492] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e: Bootstrap starting.
I20240501 19:57:01.915659 6134 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:01.915743 6134 ts_tablet_manager.cc:1367] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:01.916113 6134 raft_consensus.cc:348] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.916252 6134 raft_consensus.cc:374] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.916280 6137 tablet_bootstrap.cc:654] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.916288 6134 raft_consensus.cc:729] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.916415 6134 consensus_queue.cc:260] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.916678 6134 ts_tablet_manager.cc:1398] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.916842 6134 tablet_bootstrap.cc:492] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:01.917099 5241 tablet_service.cc:1467] Processing CreateTablet for tablet fc16a5fd751d4f109b420ed953abe053 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:01.917527 6134 tablet_bootstrap.cc:654] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.917809 6137 tablet_bootstrap.cc:492] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e: No bootstrap required, opened a new log
I20240501 19:57:01.917878 6137 ts_tablet_manager.cc:1367] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:01.918026 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc16a5fd751d4f109b420ed953abe053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:01.918254 6137 raft_consensus.cc:348] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.918352 6137 raft_consensus.cc:374] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.918416 6137 raft_consensus.cc:729] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.918560 6137 consensus_queue.cc:260] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.918937 6137 ts_tablet_manager.cc:1398] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.919865 6134 tablet_bootstrap.cc:492] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:01.919936 6134 ts_tablet_manager.cc:1367] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:01.920389 6133 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Bootstrap starting.
I20240501 19:57:01.920347 6134 raft_consensus.cc:348] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.920491 6134 raft_consensus.cc:374] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.920572 6134 raft_consensus.cc:729] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.920744 6134 consensus_queue.cc:260] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:01.921064 6134 ts_tablet_manager.cc:1398] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.921262 6134 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Bootstrap starting.
I20240501 19:57:01.921993 6134 tablet_bootstrap.cc:654] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.922398 6133 tablet_bootstrap.cc:654] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:01.923346 6133 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: No bootstrap required, opened a new log
I20240501 19:57:01.923362 6134 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: No bootstrap required, opened a new log
I20240501 19:57:01.923425 6134 ts_tablet_manager.cc:1367] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:01.923425 6133 ts_tablet_manager.cc:1367] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:57:01.923772 6134 raft_consensus.cc:348] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.923772 6133 raft_consensus.cc:348] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.923863 6133 raft_consensus.cc:374] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.923863 6134 raft_consensus.cc:374] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:01.923903 6133 raft_consensus.cc:729] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.923903 6134 raft_consensus.cc:729] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Initialized, Role: FOLLOWER
I20240501 19:57:01.924013 6133 consensus_queue.cc:260] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.924013 6134 consensus_queue.cc:260] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.924238 6133 ts_tablet_manager.cc:1398] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:01.924237 6134 ts_tablet_manager.cc:1398] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:01.946242 6127 raft_consensus.cc:482] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:01.946341 6127 raft_consensus.cc:504] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.946648 6127 leader_election.cc:288] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.946959 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:57:01.947057 5130 raft_consensus.cc:2454] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:57:01.947316 5196 leader_election.cc:302] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.947468 6127 raft_consensus.cc:2781] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:01.947525 6127 raft_consensus.cc:482] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:01.947554 6127 raft_consensus.cc:3037] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.948289 6127 raft_consensus.cc:504] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.948531 6127 leader_election.cc:288] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:01.948801 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:01.948884 5130 raft_consensus.cc:3037] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:01.949502 5130 raft_consensus.cc:2454] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:57:01.949759 5196 leader_election.cc:302] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:01.949895 6127 raft_consensus.cc:2781] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:01.949970 6127 raft_consensus.cc:686] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:01.950093 6127 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } }
I20240501 19:57:01.951273 4848 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } }
I20240501 19:57:02.009398 6127 raft_consensus.cc:482] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.009487 6127 raft_consensus.cc:504] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.009793 6127 leader_election.cc:288] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:02.010200 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0753c139624897b37cab9f3a9eab35" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" is_pre_election: true
I20240501 19:57:02.010326 5130 raft_consensus.cc:2454] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 0.
I20240501 19:57:02.010597 5196 leader_election.cc:302] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:02.010773 6127 raft_consensus.cc:2781] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.010823 6127 raft_consensus.cc:482] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.010851 6127 raft_consensus.cc:3037] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.011617 6127 raft_consensus.cc:504] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.011875 6127 leader_election.cc:288] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:02.012198 5130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b0753c139624897b37cab9f3a9eab35" candidate_uuid: "4f272a2b03c244d69ff5595d457853ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:02.012298 5130 raft_consensus.cc:3037] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.012984 5130 raft_consensus.cc:2454] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f272a2b03c244d69ff5595d457853ac in term 1.
I20240501 19:57:02.013236 5196 leader_election.cc:302] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, 4f272a2b03c244d69ff5595d457853ac; no voters:
I20240501 19:57:02.013409 6127 raft_consensus.cc:2781] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.013496 6127 raft_consensus.cc:686] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Leader. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:02.013639 6127 consensus_queue.cc:237] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.014981 4848 catalog_manager.cc:5484] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f272a2b03c244d69ff5595d457853ac (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
I20240501 19:57:02.017817 6069 raft_consensus.cc:482] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:02.017889 6069 raft_consensus.cc:504] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.018150 6069 leader_election.cc:288] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:02.018553 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbb6869820148cbb31738a0efc7c515" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e" is_pre_election: true
I20240501 19:57:02.018664 4999 raft_consensus.cc:2454] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 0.
I20240501 19:57:02.018901 5066 leader_election.cc:302] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:02.019075 6069 raft_consensus.cc:2781] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:02.019150 6069 raft_consensus.cc:482] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:02.019178 6069 raft_consensus.cc:3037] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.019884 6069 raft_consensus.cc:504] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.020107 6069 leader_election.cc:288] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:02.020515 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbb6869820148cbb31738a0efc7c515" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:02.020604 4999 raft_consensus.cc:3037] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:02.021306 4999 raft_consensus.cc:2454] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 1.
I20240501 19:57:02.021549 5066 leader_election.cc:302] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:02.021711 6069 raft_consensus.cc:2781] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:02.021878 6069 raft_consensus.cc:686] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:57:02.021988 6069 consensus_queue.cc:237] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } }
I20240501 19:57:02.023237 4848 catalog_manager.cc:5484] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66). New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } }
W20240501 19:57:02.133548 6146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:02.133726 6146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:02.136435 6146 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:57:02.215631 5130 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.215631 5129 raft_consensus.cc:1261] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.216101 6127 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.216382 6079 consensus_queue.cc:1035] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:02.220472 4999 raft_consensus.cc:1261] T ecbb6869820148cbb31738a0efc7c515 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:02.220963 6069 consensus_queue.cc:1035] T ecbb6869820148cbb31738a0efc7c515 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 90.0133 ms
time per row: 0.0450067 ms
I20240501 19:57:02.262508 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.68:34679
--local_ip_for_outbound_sockets=127.30.1.68
--tserver_master_addrs=127.30.1.126:44833
--webserver_port=46171
--webserver_interface=127.30.1.68
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:57:02.359813 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:02.359962 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:02.360028 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:02.362857 6184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:02.362906 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:02.362962 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.68
I20240501 19:57:02.366065 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.68:34679
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.1.68
--webserver_port=46171
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.68
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:02.366685 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:02.367583 6184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:02.375078 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.375497 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.376080 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:02.376859 6184 server_base.cc:1034] running on GCE node
I20240501 19:57:02.377171 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:02.377579 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:02.378729 6184 hybrid_clock.cc:648] HybridClock initialized: now 1714593422378709 us; error 30 us; skew 500 ppm
I20240501 19:57:02.380237 6184 webserver.cc:460] Webserver started at http://127.30.1.68:46171/ using document root <none> and password file <none>
I20240501 19:57:02.380686 6184 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:02.380755 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:02.383685 6184 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.385336 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.386165 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.386250 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6489e19b12b44216824eeb036c208ae5"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:57:02.386550 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:02.409806 6184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:02.410287 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:02.410418 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:02.410825 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:02.411621 6184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:02.411690 6184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.411753 6184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:02.411787 6184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.420094 6184 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.68:34679
I20240501 19:57:02.420116 6311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.68:34679 every 8 connection(s)
I20240501 19:57:02.420902 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:57:02.427593 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 6184
I20240501 19:57:02.427932 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:33827
--local_ip_for_outbound_sockets=127.30.1.69
--tserver_master_addrs=127.30.1.126:44833
--webserver_port=34267
--webserver_interface=127.30.1.69
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:02.428110 6312 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:57:02.428282 6312 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:02.428645 6312 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:57:02.429574 4856 ts_manager.cc:194] Re-registered known tserver with Master: 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679)
I20240501 19:57:02.430266 4856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.68:49081
W20240501 19:57:02.526062 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:02.526216 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:02.526253 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:02.529131 6315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:02.529183 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:02.529244 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:57:02.532560 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:33827
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=34267
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:02.533248 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:02.534148 6315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:02.541149 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.541564 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.542174 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:02.542945 6315 server_base.cc:1034] running on GCE node
I20240501 19:57:02.543251 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:02.543759 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:02.544943 6315 hybrid_clock.cc:648] HybridClock initialized: now 1714593422544918 us; error 32 us; skew 500 ppm
I20240501 19:57:02.546679 6315 webserver.cc:460] Webserver started at http://127.30.1.69:34267/ using document root <none> and password file <none>
I20240501 19:57:02.547169 6315 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:02.547245 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:02.550268 6315 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:02.551970 6329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.552933 6315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.553033 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4f5fc46abef54f898b8eccf16432f798"
format_stamp: "Formatted at 2024-05-01 19:56:54 on dist-test-slave-htmr"
I20240501 19:57:02.553395 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:02.588649 6315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:02.589252 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:02.589390 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:02.589838 6315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:02.590646 6315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:02.590692 6315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.590744 6315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:02.590776 6315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.598687 6315 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:33827
I20240501 19:57:02.598718 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:33827 every 8 connection(s)
I20240501 19:57:02.599531 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:57:02.603271 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 6315
I20240501 19:57:02.603794 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:41385
--local_ip_for_outbound_sockets=127.30.1.70
--tserver_master_addrs=127.30.1.126:44833
--webserver_port=41475
--webserver_interface=127.30.1.70
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:02.607542 6444 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:57:02.607733 6444 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:02.608094 6444 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:57:02.609051 4856 ts_manager.cc:194] Re-registered known tserver with Master: 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827)
I20240501 19:57:02.609838 4856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:33255
W20240501 19:57:02.700718 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:02.700872 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:02.700953 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:02.703718 6447 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:02.703766 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:02.703821 6447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:57:02.707077 6447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:41385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=41475
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:02.707676 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:02.708634 6447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:02.715761 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.716466 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.716958 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:02.717680 6447 server_base.cc:1034] running on GCE node
I20240501 19:57:02.717972 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:02.718429 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:02.719565 6447 hybrid_clock.cc:648] HybridClock initialized: now 1714593422719563 us; error 32 us; skew 500 ppm
I20240501 19:57:02.721236 6447 webserver.cc:460] Webserver started at http://127.30.1.70:41475/ using document root <none> and password file <none>
I20240501 19:57:02.721686 6447 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:02.721750 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:02.724598 6447 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:02.726285 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.727172 6447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:02.727254 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "eaa8d49089fc428ab6e6601637abe221"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:57:02.727576 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:02.740561 6447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:02.741044 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:02.741173 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:02.741602 6447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:02.742609 6469 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 19:57:02.745218 6447 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 19:57:02.745267 6447 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 19:57:02.745309 6447 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 19:57:02.746546 6447 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 19:57:02.746587 6447 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:02.746850 6469 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:57:02.755550 6447 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:41385
I20240501 19:57:02.755854 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:41385 every 8 connection(s)
I20240501 19:57:02.756454 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:57:02.758689 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 6447
I20240501 19:57:02.759066 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:35049
--local_ip_for_outbound_sockets=127.30.1.71
--tserver_master_addrs=127.30.1.126:44833
--webserver_port=41261
--webserver_interface=127.30.1.71
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:02.760727 6469 log.cc:824] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:02.761931 6469 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:02.762225 6469 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Bootstrap complete.
I20240501 19:57:02.763386 6469 ts_tablet_manager.cc:1367] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20240501 19:57:02.765722 6577 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:57:02.765925 6577 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:02.766424 6577 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:57:02.766172 6469 raft_consensus.cc:348] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:02.766520 6469 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:57:02.766949 6469 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:02.767659 6469 ts_tablet_manager.cc:1398] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 19:57:02.768488 4856 ts_manager.cc:194] Re-registered known tserver with Master: eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:57:02.769207 4856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:41381
I20240501 19:57:02.770738 6577 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
W20240501 19:57:02.862249 6580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:02.862424 6580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:02.862499 6580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:02.865381 6580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:02.865454 6580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:02.865558 6580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:57:02.868989 6580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:35049
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=41261
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.6580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:02.869699 6580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:02.870661 6580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:02.878415 6597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.878844 6595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:02.879514 6594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:02.880302 6580 server_base.cc:1034] running on GCE node
I20240501 19:57:02.880650 6580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:02.881142 6580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:02.882297 6580 hybrid_clock.cc:648] HybridClock initialized: now 1714593422882285 us; error 25 us; skew 500 ppm
I20240501 19:57:02.883908 6580 webserver.cc:460] Webserver started at http://127.30.1.71:41261/ using document root <none> and password file <none>
I20240501 19:57:02.884387 6580 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:02.884460 6580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:02.887461 6580 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.889214 6603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.890141 6580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:02.890235 6580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4b2c3262271245a4b1acf221c8e5110f"
format_stamp: "Formatted at 2024-05-01 19:56:55 on dist-test-slave-htmr"
I20240501 19:57:02.890586 6580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:02.917589 6580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:02.918146 6580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:02.918294 6580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:02.918802 6580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:02.919741 6580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:02.919790 6580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.919843 6580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:02.919880 6580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:02.927826 6580 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:35049
I20240501 19:57:02.927868 6716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:35049 every 8 connection(s)
I20240501 19:57:02.928738 6580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:57:02.934681 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 6580
I20240501 19:57:02.934871 30725 test_util.cc:268] Using random seed: -471301289
I20240501 19:57:02.937204 6717 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:57:02.937433 6717 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:02.937848 6717 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:57:02.938761 4856 ts_manager.cc:194] Re-registered known tserver with Master: 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049)
I20240501 19:57:02.939430 4856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:36189
I20240501 19:57:02.944495 30725 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 19:57:02.944684 30725 test_util.cc:268] Using random seed: -471291477
I20240501 19:57:02.952827 30725 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 19:57:02.952996 30725 test_util.cc:268] Using random seed: -471283166
I20240501 19:57:02.963550 30725 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 19:57:02.963692 30725 test_util.cc:268] Using random seed: -471272469
I20240501 19:57:02.975052 30725 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 19:57:02.975205 30725 test_util.cc:268] Using random seed: -471260957
I20240501 19:57:02.989820 30725 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 19:57:03.412328 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:03.428145 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.428260 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.431995 6312 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:57:03.433617 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.445087 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:03.445508 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.470916 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:03.612201 6444 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:57:03.778803 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.785193 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:03.790459 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:03.793401 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.811568 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:03.818549 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.829845 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:03.941687 6717 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:57:03.946868 4998 consensus_queue.cc:237] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:03.973961 5129 raft_consensus.cc:1261] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240501 19:57:03.978364 4933 consensus_peers.cc:485] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): Couldn't send request to peer 6489e19b12b44216824eeb036c208ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaad9f75c694c84a890194769b4b461. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:03.981658 6781 consensus_queue.cc:1035] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:03.987830 6781 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index -1 to 31, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:03.990494 4856 catalog_manager.cc:5484] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index -1 to 31, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:03.998594 5128 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index -1 to 31, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.062474 4999 consensus_queue.cc:237] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:04.063657 5259 raft_consensus.cc:1261] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240501 19:57:04.064040 5890 consensus_queue.cc:1035] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 19:57:04.066104 5890 raft_consensus.cc:2932] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.066944 5259 raft_consensus.cc:2932] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
W20240501 19:57:04.068743 4932 consensus_peers.cc:485] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e -> Peer 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049): Couldn't send request to peer 4b2c3262271245a4b1acf221c8e5110f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1656d575d7f4d79bb289c3a043cf6aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.069291 4856 catalog_manager.cc:5484] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index -1 to 38, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.149222 5260 consensus_queue.cc:237] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:04.152659 4999 raft_consensus.cc:1261] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch)
I20240501 19:57:04.153669 6795 consensus_queue.cc:1035] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20240501 19:57:04.161151 6795 raft_consensus.cc:2932] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
W20240501 19:57:04.161850 5194 consensus_peers.cc:485] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac -> Peer 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827): Couldn't send request to peer 4f5fc46abef54f898b8eccf16432f798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 259f0c2339954fa7b9a1d4e604a8511a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.163789 4848 catalog_manager.cc:5484] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 43, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.164970 4999 raft_consensus.cc:2932] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.222687 5130 consensus_queue.cc:237] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:04.224563 4999 raft_consensus.cc:1261] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240501 19:57:04.225126 6809 consensus_queue.cc:1035] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 42, Time since last communication: 0.000s
I20240501 19:57:04.226737 6181 raft_consensus.cc:2932] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:04.227454 4999 raft_consensus.cc:2932] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:04.230577 4848 catalog_manager.cc:5484] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index -1 to 44, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
W20240501 19:57:04.270249 5064 consensus_peers.cc:485] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cbdfb0387a9449f897a454ff26276b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.329733 5259 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:04.331499 5128 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 19:57:04.333250 6127 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
W20240501 19:57:04.336781 5196 consensus_peers.cc:485] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac -> Peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): Couldn't send request to peer 6489e19b12b44216824eeb036c208ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc16a5fd751d4f109b420ed953abe053. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.336879 6127 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.337512 5129 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.339205 4848 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 63, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.362129 5128 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:04.367941 5259 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240501 19:57:04.368414 6181 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
W20240501 19:57:04.369284 5063 consensus_peers.cc:485] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 -> Peer 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827): Couldn't send request to peer 4f5fc46abef54f898b8eccf16432f798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6b4cf93c1be45218a4c0bc0f7cb4c97. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.370141 6766 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.370390 5259 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.372442 4856 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index -1 to 61, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.398403 7009 ts_tablet_manager.cc:902] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:04.399236 7009 tablet_copy_client.cc:308] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:04.414490 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet aeaad9f75c694c84a890194769b4b461 from peer 6489e19b12b44216824eeb036c208ae5 ({username='slave'} at 127.30.1.68:47723)
I20240501 19:57:04.414608 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet aeaad9f75c694c84a890194769b4b461 from peer 6489e19b12b44216824eeb036c208ae5 at {username='slave'} at 127.30.1.68:47723: session id = 6489e19b12b44216824eeb036c208ae5-aeaad9f75c694c84a890194769b4b461
I20240501 19:57:04.416020 5019 tablet_copy_source_session.cc:205] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.423811 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaad9f75c694c84a890194769b4b461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.434309 7009 tablet_copy_client.cc:783] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.434547 7009 tablet_copy_client.cc:647] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.435671 5260 consensus_queue.cc:237] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:04.437629 5128 raft_consensus.cc:1261] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240501 19:57:04.438207 6127 consensus_queue.cc:1035] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20240501 19:57:04.440351 7009 tablet_copy_client.cc:523] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.441761 6586 raft_consensus.cc:2932] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.441874 7009 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Bootstrap starting.
I20240501 19:57:04.443909 4848 catalog_manager.cc:5484] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 59, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.444224 5128 raft_consensus.cc:2932] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
W20240501 19:57:04.446632 5195 consensus_peers.cc:485] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac -> Peer 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049): Couldn't send request to peer 4b2c3262271245a4b1acf221c8e5110f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b0753c139624897b37cab9f3a9eab35. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.457373 7009 log.cc:824] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:04.519043 7009 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=725 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:04.519577 7009 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Bootstrap complete.
I20240501 19:57:04.519973 7009 ts_tablet_manager.cc:1367] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Time spent bootstrapping tablet: real 0.078s user 0.060s sys 0.007s
I20240501 19:57:04.522554 7009 raft_consensus.cc:348] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:04.522843 7009 raft_consensus.cc:729] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.523336 7009 consensus_queue.cc:260] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:04.524420 7009 ts_tablet_manager.cc:1398] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:04.527402 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 6489e19b12b44216824eeb036c208ae5-aeaad9f75c694c84a890194769b4b461 received from {username='slave'} at 127.30.1.68:47723
I20240501 19:57:04.527489 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 6489e19b12b44216824eeb036c208ae5-aeaad9f75c694c84a890194769b4b461 on tablet aeaad9f75c694c84a890194769b4b461 with peer 6489e19b12b44216824eeb036c208ae5
I20240501 19:57:04.532970 5260 consensus_queue.cc:237] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
W20240501 19:57:04.534511 5195 consensus_peers.cc:485] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdb072bde014a35989544648d90249b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.539934 4998 raft_consensus.cc:1261] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 19:57:04.542121 6586 consensus_queue.cc:1035] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 19:57:04.547894 6586 raft_consensus.cc:2932] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:04.551775 4848 catalog_manager.cc:5484] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 72, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:04.557996 4998 raft_consensus.cc:2932] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:04.617218 7047 ts_tablet_manager.cc:902] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Initiating tablet copy from peer 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:04.618013 7047 tablet_copy_client.cc:308] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:43517
I20240501 19:57:04.625239 5150 tablet_copy_service.cc:140] P 43061d2e1908476eaf2b3bdc9a402ee2: Received BeginTabletCopySession request for tablet 0cbdfb0387a9449f897a454ff26276b4 from peer eaa8d49089fc428ab6e6601637abe221 ({username='slave'} at 127.30.1.70:55749)
I20240501 19:57:04.625362 5150 tablet_copy_service.cc:161] P 43061d2e1908476eaf2b3bdc9a402ee2: Beginning new tablet copy session on tablet 0cbdfb0387a9449f897a454ff26276b4 from peer eaa8d49089fc428ab6e6601637abe221 at {username='slave'} at 127.30.1.70:55749: session id = eaa8d49089fc428ab6e6601637abe221-0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:04.626783 5150 tablet_copy_source_session.cc:205] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.629607 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbdfb0387a9449f897a454ff26276b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.632530 4996 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
W20240501 19:57:04.634821 4933 consensus_peers.cc:485] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): Couldn't send request to peer 6489e19b12b44216824eeb036c208ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe3059729d0d4d7ebf4c2f39071919f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.637061 7047 tablet_copy_client.cc:783] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.637285 7047 tablet_copy_client.cc:647] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.637614 5260 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240501 19:57:04.645500 7047 tablet_copy_client.cc:523] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.646744 7047 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:57:04.647439 6781 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240501 19:57:04.649200 6781 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.651473 4848 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index -1 to 66, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.655354 5260 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:04.660768 7057 ts_tablet_manager.cc:902] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:04.663126 7057 tablet_copy_client.cc:308] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:04.685735 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet d1656d575d7f4d79bb289c3a043cf6aa from peer 4b2c3262271245a4b1acf221c8e5110f ({username='slave'} at 127.30.1.71:56179)
I20240501 19:57:04.685834 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet d1656d575d7f4d79bb289c3a043cf6aa from peer 4b2c3262271245a4b1acf221c8e5110f at {username='slave'} at 127.30.1.71:56179: session id = 4b2c3262271245a4b1acf221c8e5110f-d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:04.686691 5019 tablet_copy_source_session.cc:205] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.690088 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1656d575d7f4d79bb289c3a043cf6aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.697806 7057 tablet_copy_client.cc:783] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.698015 7057 tablet_copy_client.cc:647] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.700600 7057 tablet_copy_client.cc:523] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.704363 7057 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap starting.
I20240501 19:57:04.725695 7057 log.cc:824] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:04.733808 7047 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:04.734134 7047 tablet_bootstrap.cc:492] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Bootstrap complete.
I20240501 19:57:04.734331 7047 ts_tablet_manager.cc:1367] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.088s user 0.061s sys 0.000s
I20240501 19:57:04.734836 7047 raft_consensus.cc:348] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:04.736282 7047 raft_consensus.cc:729] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.736511 7047 consensus_queue.cc:260] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:04.736956 7047 ts_tablet_manager.cc:1398] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:04.738121 5150 tablet_copy_service.cc:342] P 43061d2e1908476eaf2b3bdc9a402ee2: Request end of tablet copy session eaa8d49089fc428ab6e6601637abe221-0cbdfb0387a9449f897a454ff26276b4 received from {username='slave'} at 127.30.1.70:55749
I20240501 19:57:04.738209 5150 tablet_copy_service.cc:433] P 43061d2e1908476eaf2b3bdc9a402ee2: ending tablet copy session eaa8d49089fc428ab6e6601637abe221-0cbdfb0387a9449f897a454ff26276b4 on tablet 0cbdfb0387a9449f897a454ff26276b4 with peer eaa8d49089fc428ab6e6601637abe221
I20240501 19:57:04.747249 4998 consensus_queue.cc:237] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:04.750983 5129 raft_consensus.cc:1261] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240501 19:57:04.751987 6782 consensus_queue.cc:1035] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
W20240501 19:57:04.757449 4931 consensus_peers.cc:485] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827): Couldn't send request to peer 4f5fc46abef54f898b8eccf16432f798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ca2013d6f6b496e99af4a248bd3ed57. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.778266 6745 raft_consensus.cc:2932] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.780718 7079 ts_tablet_manager.cc:902] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:04.781476 7079 tablet_copy_client.cc:308] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:04.783324 5126 raft_consensus.cc:2932] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.786265 4848 catalog_manager.cc:5484] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index -1 to 62, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:04.798586 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet 259f0c2339954fa7b9a1d4e604a8511a from peer 4f5fc46abef54f898b8eccf16432f798 ({username='slave'} at 127.30.1.69:51389)
I20240501 19:57:04.798727 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet 259f0c2339954fa7b9a1d4e604a8511a from peer 4f5fc46abef54f898b8eccf16432f798 at {username='slave'} at 127.30.1.69:51389: session id = 4f5fc46abef54f898b8eccf16432f798-259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:04.799559 7057 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=63 ignored=0} inserts{seen=748 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:04.799927 7057 tablet_bootstrap.cc:492] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap complete.
I20240501 19:57:04.800038 5281 tablet_copy_source_session.cc:205] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.800228 7057 ts_tablet_manager.cc:1367] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Time spent bootstrapping tablet: real 0.096s user 0.066s sys 0.008s
I20240501 19:57:04.803414 7057 raft_consensus.cc:348] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:04.804796 7057 raft_consensus.cc:729] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2c3262271245a4b1acf221c8e5110f, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.805357 7057 consensus_queue.cc:260] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:04.806219 7057 ts_tablet_manager.cc:1398] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:57:04.806376 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259f0c2339954fa7b9a1d4e604a8511a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.807353 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 4b2c3262271245a4b1acf221c8e5110f-d1656d575d7f4d79bb289c3a043cf6aa received from {username='slave'} at 127.30.1.71:56179
I20240501 19:57:04.807401 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 4b2c3262271245a4b1acf221c8e5110f-d1656d575d7f4d79bb289c3a043cf6aa on tablet d1656d575d7f4d79bb289c3a043cf6aa with peer 4b2c3262271245a4b1acf221c8e5110f
I20240501 19:57:04.822796 7079 tablet_copy_client.cc:783] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.823024 7079 tablet_copy_client.cc:647] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.828871 7079 tablet_copy_client.cc:523] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.831894 7079 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Bootstrap starting.
I20240501 19:57:04.854945 7079 log.cc:824] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:04.909982 4999 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:04.911810 5130 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 19:57:04.912730 6782 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
W20240501 19:57:04.913827 4932 consensus_peers.cc:485] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049): Couldn't send request to peer 4b2c3262271245a4b1acf221c8e5110f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3fd829e098745598ccb473e5a0503b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:04.914940 5130 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.914525 5890 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index -1 to 67, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.924527 4856 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index -1 to 67, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:04.941949 7009 ts_tablet_manager.cc:902] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:04.942391 7009 tablet_copy_client.cc:308] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:04.950250 7079 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:04.950599 7079 tablet_bootstrap.cc:492] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Bootstrap complete.
I20240501 19:57:04.950959 7079 ts_tablet_manager.cc:1367] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Time spent bootstrapping tablet: real 0.119s user 0.070s sys 0.011s
I20240501 19:57:04.956261 7079 raft_consensus.cc:348] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:04.956465 7079 raft_consensus.cc:729] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fc46abef54f898b8eccf16432f798, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.956892 7079 consensus_queue.cc:260] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:04.957870 7079 ts_tablet_manager.cc:1398] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 19:57:04.962066 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session 4f5fc46abef54f898b8eccf16432f798-259f0c2339954fa7b9a1d4e604a8511a received from {username='slave'} at 127.30.1.69:51389
I20240501 19:57:04.962162 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session 4f5fc46abef54f898b8eccf16432f798-259f0c2339954fa7b9a1d4e604a8511a on tablet 259f0c2339954fa7b9a1d4e604a8511a with peer 4f5fc46abef54f898b8eccf16432f798
I20240501 19:57:04.965198 7079 ts_tablet_manager.cc:902] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Initiating tablet copy from peer 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:04.965576 7079 tablet_copy_client.cc:308] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:43517
I20240501 19:57:04.967850 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet fc16a5fd751d4f109b420ed953abe053 from peer 6489e19b12b44216824eeb036c208ae5 ({username='slave'} at 127.30.1.68:38041)
I20240501 19:57:04.967932 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet fc16a5fd751d4f109b420ed953abe053 from peer 6489e19b12b44216824eeb036c208ae5 at {username='slave'} at 127.30.1.68:38041: session id = 6489e19b12b44216824eeb036c208ae5-fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:04.968729 5281 tablet_copy_source_session.cc:205] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.969508 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc16a5fd751d4f109b420ed953abe053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.971437 5150 tablet_copy_service.cc:140] P 43061d2e1908476eaf2b3bdc9a402ee2: Received BeginTabletCopySession request for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 from peer 4f5fc46abef54f898b8eccf16432f798 ({username='slave'} at 127.30.1.69:42783)
I20240501 19:57:04.971526 5150 tablet_copy_service.cc:161] P 43061d2e1908476eaf2b3bdc9a402ee2: Beginning new tablet copy session on tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 from peer 4f5fc46abef54f898b8eccf16432f798 at {username='slave'} at 127.30.1.69:42783: session id = 4f5fc46abef54f898b8eccf16432f798-b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:04.971626 7009 tablet_copy_client.cc:783] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.971782 7009 tablet_copy_client.cc:647] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.972268 5150 tablet_copy_source_session.cc:205] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.974092 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.974355 7057 ts_tablet_manager.cc:902] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:04.974749 7009 tablet_copy_client.cc:523] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.974771 7057 tablet_copy_client.cc:308] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:04.975858 7009 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Bootstrap starting.
I20240501 19:57:04.977288 7079 tablet_copy_client.cc:783] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.977448 7079 tablet_copy_client.cc:647] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.978868 7079 tablet_copy_client.cc:523] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:04.980353 7079 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap starting.
I20240501 19:57:04.988279 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet 9b0753c139624897b37cab9f3a9eab35 from peer 4b2c3262271245a4b1acf221c8e5110f ({username='slave'} at 127.30.1.71:53137)
I20240501 19:57:04.988358 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet 9b0753c139624897b37cab9f3a9eab35 from peer 4b2c3262271245a4b1acf221c8e5110f at {username='slave'} at 127.30.1.71:53137: session id = 4b2c3262271245a4b1acf221c8e5110f-9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:04.989180 5281 tablet_copy_source_session.cc:205] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:04.990087 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b0753c139624897b37cab9f3a9eab35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.992071 7057 tablet_copy_client.cc:783] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:04.992238 7057 tablet_copy_client.cc:647] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:04.999171 6266 raft_consensus.cc:1203] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.30->[1.31-1.72] Dedup: 1.53->[1.54-1.72]
I20240501 19:57:05.017776 7057 tablet_copy_client.cc:523] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.019447 7057 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap starting.
I20240501 19:57:05.039103 6670 raft_consensus.cc:1203] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.74] Dedup: 1.64->[1.65-1.74]
I20240501 19:57:05.072613 7079 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=711 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.072954 7079 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap complete.
I20240501 19:57:05.073228 7079 ts_tablet_manager.cc:1367] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Time spent bootstrapping tablet: real 0.093s user 0.073s sys 0.006s
I20240501 19:57:05.073781 7079 raft_consensus.cc:348] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:05.073901 7079 raft_consensus.cc:729] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fc46abef54f898b8eccf16432f798, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.074059 7079 consensus_queue.cc:260] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:05.074365 7079 ts_tablet_manager.cc:1398] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:05.074815 5150 tablet_copy_service.cc:342] P 43061d2e1908476eaf2b3bdc9a402ee2: Request end of tablet copy session 4f5fc46abef54f898b8eccf16432f798-b6b4cf93c1be45218a4c0bc0f7cb4c97 received from {username='slave'} at 127.30.1.69:42783
I20240501 19:57:05.074883 5150 tablet_copy_service.cc:433] P 43061d2e1908476eaf2b3bdc9a402ee2: ending tablet copy session 4f5fc46abef54f898b8eccf16432f798-b6b4cf93c1be45218a4c0bc0f7cb4c97 on tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 with peer 4f5fc46abef54f898b8eccf16432f798
I20240501 19:57:05.121886 7047 ts_tablet_manager.cc:902] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:05.122349 7047 tablet_copy_client.cc:308] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:05.127133 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet 1fdb072bde014a35989544648d90249b from peer eaa8d49089fc428ab6e6601637abe221 ({username='slave'} at 127.30.1.70:59581)
I20240501 19:57:05.127233 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet 1fdb072bde014a35989544648d90249b from peer eaa8d49089fc428ab6e6601637abe221 at {username='slave'} at 127.30.1.70:59581: session id = eaa8d49089fc428ab6e6601637abe221-1fdb072bde014a35989544648d90249b
I20240501 19:57:05.128177 5281 tablet_copy_source_session.cc:205] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:05.130497 6531 raft_consensus.cc:1203] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.66] Dedup: 1.51->[1.52-1.66]
I20240501 19:57:05.133407 7009 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=715 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.133589 5260 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.133672 7009 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Bootstrap complete.
I20240501 19:57:05.133838 7009 ts_tablet_manager.cc:1367] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Time spent bootstrapping tablet: real 0.158s user 0.082s sys 0.004s
I20240501 19:57:05.134577 7009 raft_consensus.cc:348] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.134714 7009 raft_consensus.cc:729] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.134910 7009 consensus_queue.cc:260] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.135283 6530 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:57:05.135295 5196 consensus_peers.cc:485] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac -> Peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): Couldn't send request to peer 6489e19b12b44216824eeb036c208ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a054a3f0d0ca40d98979620a3bf39a1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:05.135843 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb072bde014a35989544648d90249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.136737 7009 ts_tablet_manager.cc:1398] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:05.137806 7047 tablet_copy_client.cc:783] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:05.138024 7047 tablet_copy_client.cc:647] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:05.138409 7103 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:57:05.140734 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session 6489e19b12b44216824eeb036c208ae5-fc16a5fd751d4f109b420ed953abe053 received from {username='slave'} at 127.30.1.68:38041
I20240501 19:57:05.140794 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session 6489e19b12b44216824eeb036c208ae5-fc16a5fd751d4f109b420ed953abe053 on tablet fc16a5fd751d4f109b420ed953abe053 with peer 6489e19b12b44216824eeb036c208ae5
I20240501 19:57:05.144837 7047 tablet_copy_client.cc:523] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.145949 7047 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:57:05.166220 7009 ts_tablet_manager.cc:902] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:05.166631 7009 tablet_copy_client.cc:308] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:05.167078 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet fe3059729d0d4d7ebf4c2f39071919f6 from peer 6489e19b12b44216824eeb036c208ae5 ({username='slave'} at 127.30.1.68:47723)
I20240501 19:57:05.167160 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet fe3059729d0d4d7ebf4c2f39071919f6 from peer 6489e19b12b44216824eeb036c208ae5 at {username='slave'} at 127.30.1.68:47723: session id = 6489e19b12b44216824eeb036c208ae5-fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:05.167768 5019 tablet_copy_source_session.cc:205] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:05.169327 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3059729d0d4d7ebf4c2f39071919f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.172847 6820 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:05.175426 6530 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:05.177043 4856 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 2, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:05.188767 7009 tablet_copy_client.cc:783] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:05.189319 7009 tablet_copy_client.cc:647] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:05.194120 7009 tablet_copy_client.cc:523] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.195261 7009 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Bootstrap starting.
I20240501 19:57:05.262526 7057 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=87 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.263303 7057 tablet_bootstrap.cc:492] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap complete.
I20240501 19:57:05.263564 7057 ts_tablet_manager.cc:1367] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Time spent bootstrapping tablet: real 0.244s user 0.080s sys 0.004s
I20240501 19:57:05.264156 7057 raft_consensus.cc:348] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:05.264343 7057 raft_consensus.cc:729] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2c3262271245a4b1acf221c8e5110f, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.264570 7057 consensus_queue.cc:260] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:05.266439 7057 ts_tablet_manager.cc:1398] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:05.269551 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session 4b2c3262271245a4b1acf221c8e5110f-9b0753c139624897b37cab9f3a9eab35 received from {username='slave'} at 127.30.1.71:53137
I20240501 19:57:05.269629 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session 4b2c3262271245a4b1acf221c8e5110f-9b0753c139624897b37cab9f3a9eab35 on tablet 9b0753c139624897b37cab9f3a9eab35 with peer 4b2c3262271245a4b1acf221c8e5110f
I20240501 19:57:05.282004 7047 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Bootstrap replayed 1/1 log segments. Stats: ops{read=101 overwritten=0 applied=101 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.282367 7047 tablet_bootstrap.cc:492] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Bootstrap complete.
I20240501 19:57:05.282569 7047 ts_tablet_manager.cc:1367] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.137s user 0.091s sys 0.000s
I20240501 19:57:05.282974 7047 raft_consensus.cc:348] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:05.283133 7047 raft_consensus.cc:729] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.283485 7047 consensus_queue.cc:260] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.101, Last appended by leader: 101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:05.283787 7047 ts_tablet_manager.cc:1398] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.286588 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session eaa8d49089fc428ab6e6601637abe221-1fdb072bde014a35989544648d90249b received from {username='slave'} at 127.30.1.70:59581
I20240501 19:57:05.286660 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session eaa8d49089fc428ab6e6601637abe221-1fdb072bde014a35989544648d90249b on tablet 1fdb072bde014a35989544648d90249b with peer eaa8d49089fc428ab6e6601637abe221
I20240501 19:57:05.326462 7057 ts_tablet_manager.cc:902] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:05.326844 7057 tablet_copy_client.cc:308] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:05.328048 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet a3fd829e098745598ccb473e5a0503b0 from peer 4b2c3262271245a4b1acf221c8e5110f ({username='slave'} at 127.30.1.71:56179)
I20240501 19:57:05.328156 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet a3fd829e098745598ccb473e5a0503b0 from peer 4b2c3262271245a4b1acf221c8e5110f at {username='slave'} at 127.30.1.71:56179: session id = 4b2c3262271245a4b1acf221c8e5110f-a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:05.329025 5019 tablet_copy_source_session.cc:205] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:05.331357 7057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd829e098745598ccb473e5a0503b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.333648 6398 raft_consensus.cc:1203] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.78] Dedup: 1.65->[1.66-1.78]
I20240501 19:57:05.335810 7057 tablet_copy_client.cc:783] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:05.335944 7057 tablet_copy_client.cc:647] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:05.339975 7057 tablet_copy_client.cc:523] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.341065 7057 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap starting.
I20240501 19:57:05.348641 7079 ts_tablet_manager.cc:902] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:05.349141 7079 tablet_copy_client.cc:308] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:05.364878 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet 0ca2013d6f6b496e99af4a248bd3ed57 from peer 4f5fc46abef54f898b8eccf16432f798 ({username='slave'} at 127.30.1.69:58879)
I20240501 19:57:05.364984 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet 0ca2013d6f6b496e99af4a248bd3ed57 from peer 4f5fc46abef54f898b8eccf16432f798 at {username='slave'} at 127.30.1.69:58879: session id = 4f5fc46abef54f898b8eccf16432f798-0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:05.365722 5019 tablet_copy_source_session.cc:205] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:05.367226 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca2013d6f6b496e99af4a248bd3ed57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.369271 7079 tablet_copy_client.cc:783] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:05.369410 7079 tablet_copy_client.cc:647] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:05.384843 7009 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=711 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.385169 7009 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Bootstrap complete.
I20240501 19:57:05.385370 7009 ts_tablet_manager.cc:1367] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Time spent bootstrapping tablet: real 0.190s user 0.077s sys 0.005s
I20240501 19:57:05.385805 7009 raft_consensus.cc:348] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.385975 7009 raft_consensus.cc:729] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.386126 7009 consensus_queue.cc:260] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.388149 7009 ts_tablet_manager.cc:1398] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:05.389102 7079 tablet_copy_client.cc:523] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.390198 7079 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap starting.
I20240501 19:57:05.392313 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 6489e19b12b44216824eeb036c208ae5-fe3059729d0d4d7ebf4c2f39071919f6 received from {username='slave'} at 127.30.1.68:47723
I20240501 19:57:05.392374 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 6489e19b12b44216824eeb036c208ae5-fe3059729d0d4d7ebf4c2f39071919f6 on tablet fe3059729d0d4d7ebf4c2f39071919f6 with peer 6489e19b12b44216824eeb036c208ae5
I20240501 19:57:05.410928 6670 raft_consensus.cc:1203] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.100] Dedup: 1.87->[1.88-1.100]
I20240501 19:57:05.438477 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:05.445614 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:05.472551 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.486274 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:05.498608 6398 raft_consensus.cc:1203] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.92] Dedup: 1.80->[1.81-1.92]
I20240501 19:57:05.520339 6266 raft_consensus.cc:1203] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.100] Dedup: 1.84->[1.85-1.100]
I20240501 19:57:05.531399 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.545341 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.554088 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.573601 6530 raft_consensus.cc:1203] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.114] Dedup: 1.101->[1.102-1.114]
I20240501 19:57:05.620682 7079 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=759 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.621017 7079 tablet_bootstrap.cc:492] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Bootstrap complete.
I20240501 19:57:05.621214 7079 ts_tablet_manager.cc:1367] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Time spent bootstrapping tablet: real 0.231s user 0.071s sys 0.007s
I20240501 19:57:05.621673 7079 raft_consensus.cc:348] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:05.621798 7079 raft_consensus.cc:729] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fc46abef54f898b8eccf16432f798, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.621922 7079 consensus_queue.cc:260] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:05.622771 7079 ts_tablet_manager.cc:1398] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:05.630062 6266 raft_consensus.cc:1203] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.65->[1.66-1.96] Dedup: 1.80->[1.81-1.96]
I20240501 19:57:05.642103 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 4f5fc46abef54f898b8eccf16432f798-0ca2013d6f6b496e99af4a248bd3ed57 received from {username='slave'} at 127.30.1.69:58879
I20240501 19:57:05.642211 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 4f5fc46abef54f898b8eccf16432f798-0ca2013d6f6b496e99af4a248bd3ed57 on tablet 0ca2013d6f6b496e99af4a248bd3ed57 with peer 4f5fc46abef54f898b8eccf16432f798
I20240501 19:57:05.653462 7009 ts_tablet_manager.cc:902] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:05.653896 7009 tablet_copy_client.cc:308] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:05.654479 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet a054a3f0d0ca40d98979620a3bf39a1d from peer 6489e19b12b44216824eeb036c208ae5 ({username='slave'} at 127.30.1.68:38041)
I20240501 19:57:05.654567 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet a054a3f0d0ca40d98979620a3bf39a1d from peer 6489e19b12b44216824eeb036c208ae5 at {username='slave'} at 127.30.1.68:38041: session id = 6489e19b12b44216824eeb036c208ae5-a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:05.655400 5281 tablet_copy_source_session.cc:205] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:05.657069 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a054a3f0d0ca40d98979620a3bf39a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.659054 7009 tablet_copy_client.cc:783] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:05.659201 7009 tablet_copy_client.cc:647] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:05.661947 7009 tablet_copy_client.cc:523] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:05.663067 7009 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Bootstrap starting.
I20240501 19:57:05.675809 7009 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:05.676036 7009 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Bootstrap complete.
I20240501 19:57:05.676221 7009 ts_tablet_manager.cc:1367] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240501 19:57:05.676614 7009 raft_consensus.cc:348] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.676723 7009 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.676867 7009 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:05.678647 7009 ts_tablet_manager.cc:1398] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:05.680321 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session 6489e19b12b44216824eeb036c208ae5-a054a3f0d0ca40d98979620a3bf39a1d received from {username='slave'} at 127.30.1.68:38041
I20240501 19:57:05.680392 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session 6489e19b12b44216824eeb036c208ae5-a054a3f0d0ca40d98979620a3bf39a1d on tablet a054a3f0d0ca40d98979620a3bf39a1d with peer 6489e19b12b44216824eeb036c208ae5
I20240501 19:57:05.714728 7057 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=76 ignored=0} inserts{seen=748 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:05.715006 7057 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap complete.
I20240501 19:57:05.715183 7057 ts_tablet_manager.cc:1367] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Time spent bootstrapping tablet: real 0.374s user 0.068s sys 0.004s
I20240501 19:57:05.715581 7057 raft_consensus.cc:348] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:05.716073 7057 raft_consensus.cc:729] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2c3262271245a4b1acf221c8e5110f, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.716238 7057 consensus_queue.cc:260] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:05.717031 7057 ts_tablet_manager.cc:1398] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:05.717705 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 4b2c3262271245a4b1acf221c8e5110f-a3fd829e098745598ccb473e5a0503b0 received from {username='slave'} at 127.30.1.71:56179
I20240501 19:57:05.717772 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 4b2c3262271245a4b1acf221c8e5110f-a3fd829e098745598ccb473e5a0503b0 on tablet a3fd829e098745598ccb473e5a0503b0 with peer 4b2c3262271245a4b1acf221c8e5110f
I20240501 19:57:05.896838 6397 raft_consensus.cc:1203] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.61->[1.62-1.87] Dedup: 1.74->[1.75-1.87]
I20240501 19:57:05.922464 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.924042 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:05.941498 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.942891 6670 raft_consensus.cc:1203] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.92] Dedup: 1.77->[1.78-1.92]
I20240501 19:57:05.944144 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:05.961828 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:05.988973 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:06.008212 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:06.250672 6265 raft_consensus.cc:1203] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:57:06.361295 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:06.363333 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:06.372251 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:06.385169 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:06.386994 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:06.416174 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:06.461654 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:06.578131 4996 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:06.579166 6265 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240501 19:57:06.579912 6745 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240501 19:57:06.581993 7104 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 66 to 125, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:06.582899 6265 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 66 to 125, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:06.585894 4856 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 66 to 125, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:06.743361 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3059729d0d4d7ebf4c2f39071919f6 with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:06.750551 7361 tablet_replica.cc:330] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:06.750978 7361 raft_consensus.cc:2229] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:06.751266 7361 raft_consensus.cc:2258] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:06.754061 7361 ts_tablet_manager.cc:1875] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:06.760622 7361 ts_tablet_manager.cc:1888] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.124
I20240501 19:57:06.760710 7361 log.cc:1196] T fe3059729d0d4d7ebf4c2f39071919f6 P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:06.761302 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet fe3059729d0d4d7ebf4c2f39071919f6 (table rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]) successfully deleted
I20240501 19:57:07.034940 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:07.046012 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:07.048272 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.053097 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.060459 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.101161 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.125435 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:07.276392 5261 consensus_queue.cc:237] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.178, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:07.278424 6531 raft_consensus.cc:1261] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240501 19:57:07.279142 7103 consensus_queue.cc:1035] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 177, Time since last communication: 0.000s
I20240501 19:57:07.280805 6820 raft_consensus.cc:2932] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.179: config changed from index 72 to 179, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:07.281430 6531 raft_consensus.cc:2932] T 1fdb072bde014a35989544648d90249b P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 72 to 179, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:07.286399 4849 catalog_manager.cc:5484] T 1fdb072bde014a35989544648d90249b P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index 72 to 179, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:07.313956 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb072bde014a35989544648d90249b with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 179) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:07.325719 7431 tablet_replica.cc:330] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:07.338968 7431 raft_consensus.cc:2229] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.339183 7431 pending_rounds.cc:62] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Trying to abort 1 pending ops.
I20240501 19:57:07.339303 7431 pending_rounds.cc:68] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Aborting op as it isn't in flight: id { term: 1 index: 178 } timestamp: 7022974678119342080 op_type: WRITE_OP write_request { tablet_id: "1fdb072bde014a35989544648d90249b" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\275V KmA\370\350\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8fba80973aa08324" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974678084603904 authz_token { token_data: "\010\272\277\312\261\006\"3\n\005slave\022*\n 4dc6b2489e674228af5b095d4efdd26b\020\001\030\001 \001(\001" signature: "<redacted>""\022\217^8y<\023\030fS\243B\036\353T9*{\252\267kZ#A\310\313\275\311\335\350\322\000z\255j\273{\r\230\222pr\365\233!~=\0266t\257S\031\341\340n\037\212\032\324\007\353\006\320" signing_key_seq_num: 0 } } request_id { client_id: "942efbd7db984fc78267f764747f6926" seq_no: 460 first_incomplete_seq_no: 460 attempt_no: 0 }
I20240501 19:57:07.342468 7431 raft_consensus.cc:2258] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.347374 7431 ts_tablet_manager.cc:1875] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:07.350479 7431 ts_tablet_manager.cc:1888] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.178
I20240501 19:57:07.350601 7431 log.cc:1196] T 1fdb072bde014a35989544648d90249b P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/1fdb072bde014a35989544648d90249b
I20240501 19:57:07.351358 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet 1fdb072bde014a35989544648d90249b (table rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]) successfully deleted
I20240501 19:57:07.573618 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.580863 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:07.596197 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:07.615602 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.616484 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.641381 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:07.674583 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:07.834086 4996 consensus_queue.cc:237] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 1.165, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:07.836966 6669 raft_consensus.cc:1261] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch)
I20240501 19:57:07.839746 6745 consensus_queue.cc:1035] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 164, Time since last communication: 0.000s
I20240501 19:57:07.842391 5890 raft_consensus.cc:2932] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.166: config changed from index 38 to 166, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:07.845319 4848 catalog_manager.cc:5484] T d1656d575d7f4d79bb289c3a043cf6aa P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 38 to 166, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:07.847882 6669 raft_consensus.cc:2932] T d1656d575d7f4d79bb289c3a043cf6aa P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 38 to 166, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:07.867121 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet d1656d575d7f4d79bb289c3a043cf6aa with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 166) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:07.874528 7494 tablet_replica.cc:330] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:07.875692 7494 raft_consensus.cc:2229] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:07.876091 7494 raft_consensus.cc:2258] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:07.877602 7494 ts_tablet_manager.cc:1875] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:07.879927 7494 ts_tablet_manager.cc:1888] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.165
I20240501 19:57:07.884620 7494 log.cc:1196] T d1656d575d7f4d79bb289c3a043cf6aa P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:07.885663 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet d1656d575d7f4d79bb289c3a043cf6aa (table rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]) successfully deleted
I20240501 19:57:08.076064 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:08.077044 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.087611 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:08.099550 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.108250 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.113377 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.143915 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:08.300949 4999 consensus_queue.cc:237] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:08.302824 6397 raft_consensus.cc:1261] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 1 index: 178. (index mismatch)
I20240501 19:57:08.303457 7104 consensus_queue.cc:1035] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 176, Time since last communication: 0.000s
I20240501 19:57:08.305205 7182 raft_consensus.cc:2932] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.178: config changed from index 62 to 178, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:08.309000 4848 catalog_manager.cc:5484] T 0ca2013d6f6b496e99af4a248bd3ed57 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 62 to 178, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:08.308957 6397 raft_consensus.cc:2932] T 0ca2013d6f6b496e99af4a248bd3ed57 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 62 to 178, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:08.314857 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca2013d6f6b496e99af4a248bd3ed57 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 178) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:08.316567 7553 tablet_replica.cc:330] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:08.317188 7553 raft_consensus.cc:2229] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.318094 7553 pending_rounds.cc:62] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Trying to abort 1 pending ops.
I20240501 19:57:08.318223 7553 pending_rounds.cc:68] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Aborting op as it isn't in flight: id { term: 1 index: 177 } timestamp: 7022974682285252608 op_type: WRITE_OP write_request { tablet_id: "0ca2013d6f6b496e99af4a248bd3ed57" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\000\214\034e\311\227\344<\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4bcf75b542718843" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974682244939776 authz_token { token_data: "\010\272\277\312\261\006\"3\n\005slave\022*\n a322394687f54ba583f06778d3760b8b\020\001\030\001 \001(\001" signature: "<redacted>""s\300\014\304`x\244\277yE\264A\224w\365\037~\203h0\341\271\216\300\262KP\323ZR\301[3u\321\360R\010\363\363\3065^\207\304\342\275\024Srb\366Y\374\361\036#\361\234\026\201\371\236H" signing_key_seq_num: 0 } } request_id { client_id: "86e30348c3f94b539611030bdd3fe4c8" seq_no: 514 first_incomplete_seq_no: 514 attempt_no: 0 }
I20240501 19:57:08.318720 7553 raft_consensus.cc:2258] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.322585 7553 ts_tablet_manager.cc:1875] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:08.326756 7553 ts_tablet_manager.cc:1888] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.177
I20240501 19:57:08.326898 7553 log.cc:1196] T 0ca2013d6f6b496e99af4a248bd3ed57 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:08.340128 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet 0ca2013d6f6b496e99af4a248bd3ed57 (table rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]) successfully deleted
I20240501 19:57:08.575353 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.582340 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.602653 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.614943 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:08.631513 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:08.645294 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:08.663949 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:08.816550 4998 consensus_queue.cc:237] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 1.162, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:08.818015 6264 raft_consensus.cc:1261] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 163. (index mismatch)
I20240501 19:57:08.818812 6745 consensus_queue.cc:1035] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 19:57:08.822460 6745 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.163: config changed from index 31 to 163, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:08.824843 4855 catalog_manager.cc:5484] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 31 to 163, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:08.824887 6264 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.163: config changed from index 31 to 163, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:08.838248 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet aeaad9f75c694c84a890194769b4b461 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 163) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:08.844453 7626 tablet_replica.cc:330] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:08.844621 7626 raft_consensus.cc:2229] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:08.844779 7626 raft_consensus.cc:2258] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:08.845553 7626 ts_tablet_manager.cc:1875] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:08.847951 7626 ts_tablet_manager.cc:1888] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.162
I20240501 19:57:08.848083 7626 log.cc:1196] T aeaad9f75c694c84a890194769b4b461 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/aeaad9f75c694c84a890194769b4b461
I20240501 19:57:08.848733 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet aeaad9f75c694c84a890194769b4b461 (table rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]) successfully deleted
I20240501 19:57:08.998517 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.004998 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.009276 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:09.009507 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.024082 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.030423 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:09.037918 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:09.174834 5261 consensus_queue.cc:237] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 1.196, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:09.176796 6396 raft_consensus.cc:1261] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 1 index: 198. (index mismatch)
I20240501 19:57:09.177484 6127 consensus_queue.cc:1035] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240501 19:57:09.179589 6820 raft_consensus.cc:2932] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.197: config changed from index 43 to 197, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.182260 4856 catalog_manager.cc:5484] T 259f0c2339954fa7b9a1d4e604a8511a P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index 43 to 197, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.183141 6398 raft_consensus.cc:2932] T 259f0c2339954fa7b9a1d4e604a8511a P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Committing config change with OpId 1.197: config changed from index 43 to 197, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 197 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.187768 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet 259f0c2339954fa7b9a1d4e604a8511a with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 197) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:09.188522 7678 tablet_replica.cc:330] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:09.188649 7678 raft_consensus.cc:2229] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:09.188820 7678 raft_consensus.cc:2258] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.189962 7678 ts_tablet_manager.cc:1875] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:09.194070 7678 ts_tablet_manager.cc:1888] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.196
I20240501 19:57:09.194197 7678 log.cc:1196] T 259f0c2339954fa7b9a1d4e604a8511a P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:09.203299 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet 259f0c2339954fa7b9a1d4e604a8511a (table rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]) successfully deleted
I20240501 19:57:09.562434 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:09.594668 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:09.597393 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.599344 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:09.608767 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.624331 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.634987 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:09.870898 5128 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 1.183, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:09.871942 6398 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240501 19:57:09.872445 7453 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240501 19:57:09.873981 7166 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Committing config change with OpId 1.184: config changed from index 61 to 184, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.874465 6396 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index 61 to 184, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.877333 4856 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index 61 to 184, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:09.883761 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 184) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:09.885103 7753 tablet_replica.cc:330] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:09.885283 7753 raft_consensus.cc:2229] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:09.885406 7753 raft_consensus.cc:2258] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:09.886085 7753 ts_tablet_manager.cc:1875] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:09.888522 7753 ts_tablet_manager.cc:1888] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.183
I20240501 19:57:09.889293 7753 log.cc:1196] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:09.890707 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 (table rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]) successfully deleted
I20240501 19:57:10.077172 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:10.086234 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.096295 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:10.098839 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:10.112740 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.113111 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.113919 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.442152 4996 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:10.456574 6671 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 286. (index mismatch)
I20240501 19:57:10.458530 7182 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 283, Time since last communication: 0.000s
I20240501 19:57:10.464529 6745 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 67 to 286, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:10.467923 4849 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 67 to 286, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:10.470342 6670 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 67 to 286, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:10.482537 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fd829e098745598ccb473e5a0503b0 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 286) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:10.505256 7815 tablet_replica.cc:330] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:10.505537 7815 raft_consensus.cc:2229] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:10.505685 7815 raft_consensus.cc:2258] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:10.507869 7815 ts_tablet_manager.cc:1875] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:10.509979 7815 ts_tablet_manager.cc:1888] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.285
I20240501 19:57:10.510104 7815 log.cc:1196] T a3fd829e098745598ccb473e5a0503b0 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:10.511456 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet a3fd829e098745598ccb473e5a0503b0 (table rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]) successfully deleted
I20240501 19:57:10.847640 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.853510 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:10.867183 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.886040 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:10.891125 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:10.904873 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:10.953877 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.132723 5260 consensus_queue.cc:237] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 264, Committed index: 264, Last appended: 1.264, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:11.134001 6670 raft_consensus.cc:1261] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 264. Preceding OpId from leader: term: 1 index: 265. (index mismatch)
I20240501 19:57:11.135174 7103 consensus_queue.cc:1035] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 265, Last known committed idx: 264, Time since last communication: 0.000s
I20240501 19:57:11.138038 7103 raft_consensus.cc:2932] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.265: config changed from index 59 to 265, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:11.138484 6670 raft_consensus.cc:2932] T 9b0753c139624897b37cab9f3a9eab35 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Committing config change with OpId 1.265: config changed from index 59 to 265, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:11.140695 4849 catalog_manager.cc:5484] T 9b0753c139624897b37cab9f3a9eab35 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index 59 to 265, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 265 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:11.152971 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b0753c139624897b37cab9f3a9eab35 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 265) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:11.153502 7888 tablet_replica.cc:330] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:11.153666 7888 raft_consensus.cc:2229] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.153857 7888 raft_consensus.cc:2258] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.155138 7888 ts_tablet_manager.cc:1875] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.167137 7888 ts_tablet_manager.cc:1888] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.264
I20240501 19:57:11.167373 7888 log.cc:1196] T 9b0753c139624897b37cab9f3a9eab35 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:11.167973 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet 9b0753c139624897b37cab9f3a9eab35 (table rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]) successfully deleted
I20240501 19:57:11.342051 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:11.344236 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.349094 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:11.354622 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.393190 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.409111 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.457245 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:11.576614 5128 consensus_queue.cc:237] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.294, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:11.578886 6530 raft_consensus.cc:1261] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 295. (index mismatch)
I20240501 19:57:11.581099 7452 consensus_queue.cc:1035] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 295, Last known committed idx: 294, Time since last communication: 0.000s
I20240501 19:57:11.585316 7717 raft_consensus.cc:2932] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Committing config change with OpId 1.295: config changed from index 44 to 295, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:11.587920 4856 catalog_manager.cc:5484] T 0cbdfb0387a9449f897a454ff26276b4 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index 44 to 295, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:11.590263 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cbdfb0387a9449f897a454ff26276b4 with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 295) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:11.590655 7940 tablet_replica.cc:330] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:11.591084 7940 raft_consensus.cc:2229] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.591243 7940 raft_consensus.cc:2258] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.592065 7940 ts_tablet_manager.cc:1875] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.595592 7940 ts_tablet_manager.cc:1888] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.294
I20240501 19:57:11.595721 7940 log.cc:1196] T 0cbdfb0387a9449f897a454ff26276b4 P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:11.597091 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet 0cbdfb0387a9449f897a454ff26276b4 (table rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]) successfully deleted
I20240501 19:57:11.597330 6530 raft_consensus.cc:2932] T 0cbdfb0387a9449f897a454ff26276b4 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Committing config change with OpId 1.295: config changed from index 44 to 295, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 295 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:11.766099 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.767316 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:11.782631 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.782804 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:11.820376 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.826098 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:11.837352 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:11.958452 5260 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:11.960137 6266 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 1 index: 307. (index mismatch)
I20240501 19:57:11.960790 6127 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s
I20240501 19:57:11.963704 6795 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.307: config changed from index 63 to 307, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:11.966122 4849 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index 63 to 307, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:11.968123 6266 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.307: config changed from index 63 to 307, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 307 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:11.970028 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet fc16a5fd751d4f109b420ed953abe053 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 307) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:11.977394 8001 tablet_replica.cc:330] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:11.977576 8001 raft_consensus.cc:2229] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:11.977712 8001 raft_consensus.cc:2258] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:11.978328 8001 ts_tablet_manager.cc:1875] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:11.980561 8001 ts_tablet_manager.cc:1888] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.306
I20240501 19:57:11.980690 8001 log.cc:1196] T fc16a5fd751d4f109b420ed953abe053 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:11.986145 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet fc16a5fd751d4f109b420ed953abe053 (table rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]) successfully deleted
I20240501 19:57:12.306902 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:12.311690 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.315830 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.322222 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:12.354825 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:12.356976 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.387712 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.466018 5260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d"
dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52810
I20240501 19:57:12.466197 5260 raft_consensus.cc:595] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:12.640941 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.646031 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.646219 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:12.658298 6246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:12.659097 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:12.677420 5241 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:12.719419 6795 raft_consensus.cc:979] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: : Instructing follower 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:12.719518 6795 raft_consensus.cc:1067] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Signalling peer 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:12.720315 6266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d"
dest_uuid: "6489e19b12b44216824eeb036c208ae5"
from {username='slave'} at 127.30.1.67:52057
I20240501 19:57:12.720830 6266 raft_consensus.cc:482] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:12.721016 6266 raft_consensus.cc:3037] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:12.722213 6266 raft_consensus.cc:504] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:12.724493 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:12.724607 5260 raft_consensus.cc:3032] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:12.724658 5260 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:12.724799 5260 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:12.724948 5260 raft_consensus.cc:3037] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:12.725914 5260 raft_consensus.cc:2454] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:12.726215 6202 leader_election.cc:302] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, 6489e19b12b44216824eeb036c208ae5; no voters:
I20240501 19:57:12.726958 6266 leader_election.cc:288] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361), eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:57:12.727911 8086 raft_consensus.cc:2781] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:12.728117 8086 raft_consensus.cc:686] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Running, Role: LEADER
I20240501 19:57:12.728475 8086 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:12.730728 4849 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 reported cstate change: term changed from 1 to 2, leader changed from 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) to 6489e19b12b44216824eeb036c208ae5 (127.30.1.68). New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:12.769754 6531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a054a3f0d0ca40d98979620a3bf39a1d" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "eaa8d49089fc428ab6e6601637abe221"
I20240501 19:57:12.769950 6531 raft_consensus.cc:3037] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:12.771052 6531 raft_consensus.cc:2454] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:12.778496 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.143728 6531 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:13.144197 8087 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:13.155442 5260 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:57:13.156386 8087 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:57:13.223449 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:13.232460 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:13.236100 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.251581 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.260032 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:13.286552 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:13.326282 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.431814 6265 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:13.433432 6531 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:57:13.434269 8087 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:13.435848 8086 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:13.438447 4850 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 2 to 4, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:13.441936 6531 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:13.475173 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet a054a3f0d0ca40d98979620a3bf39a1d with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:13.477084 8219 tablet_replica.cc:330] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:13.477205 8219 raft_consensus.cc:2229] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:13.477301 8219 raft_consensus.cc:2258] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:13.477766 8219 ts_tablet_manager.cc:1875] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:13.480422 8219 ts_tablet_manager.cc:1888] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 19:57:13.480489 8219 log.cc:1196] T a054a3f0d0ca40d98979620a3bf39a1d P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:13.482069 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet a054a3f0d0ca40d98979620a3bf39a1d (table TestTable [id=c74331f516e2486c896a4f5b5b485c80]) successfully deleted
I20240501 19:57:13.941813 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:13.942879 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.964807 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:13.974669 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:13.978579 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:13.997627 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:14.002465 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:14.131894 5128 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 1.354, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:14.134063 6397 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
W20240501 19:57:14.134213 5064 consensus_peers.cc:485] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6b4cf93c1be45218a4c0bc0f7cb4c97. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:14.134379 7717 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240501 19:57:14.136086 7717 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Committing config change with OpId 1.355: config changed from index 184 to 355, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.137887 6396 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 184 to 355, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.138685 4854 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index 184 to 355, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.201627 5260 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
W20240501 19:57:14.202720 5194 consensus_peers.cc:485] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac -> Peer 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827): Couldn't send request to peer 4f5fc46abef54f898b8eccf16432f798. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfb7ca52d5ca4789888fe71d8b29b65f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:14.205063 5128 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240501 19:57:14.205539 7103 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240501 19:57:14.209108 8222 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.384: config changed from index -1 to 384, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:14.210609 5128 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index -1 to 384, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:14.211453 4854 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index -1 to 384, VOTER 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:14.326500 4996 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:14.328034 6265 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20240501 19:57:14.328702 8274 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240501 19:57:14.330646 7858 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 125 to 322, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:14.333410 4850 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 125 to 322, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:14.332836 6265 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 125 to 322, VOTER 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71) added. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
W20240501 19:57:14.336841 4932 consensus_peers.cc:485] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049): Couldn't send request to peer 4b2c3262271245a4b1acf221c8e5110f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe3059729d0d4d7ebf4c2f39071919f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:14.396453 4996 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 1.424, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:14.399189 6669 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 424. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
W20240501 19:57:14.399299 4933 consensus_peers.cc:485] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): Couldn't send request to peer 6489e19b12b44216824eeb036c208ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3fd829e098745598ccb473e5a0503b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:14.400274 8274 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I20240501 19:57:14.404314 8274 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.425: config changed from index 286 to 425, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:14.404982 6670 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Committing config change with OpId 1.425: config changed from index 286 to 425, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New config: { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:14.407328 4850 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 286 to 425, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:14.522245 5261 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 409, Committed index: 409, Last appended: 1.409, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:14.523556 6265 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer 4f272a2b03c244d69ff5595d457853ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 409. Preceding OpId from leader: term: 1 index: 410. (index mismatch)
I20240501 19:57:14.524235 6586 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 410, Last known committed idx: 409, Time since last communication: 0.000s
W20240501 19:57:14.524677 5195 consensus_peers.cc:485] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac -> Peer eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): Couldn't send request to peer eaa8d49089fc428ab6e6601637abe221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc16a5fd751d4f109b420ed953abe053. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:14.526914 7103 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Committing config change with OpId 1.410: config changed from index 307 to 410, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.530203 4854 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac reported cstate change: config changed from index 307 to 410, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "4f272a2b03c244d69ff5595d457853ac" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.532506 6264 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.410: config changed from index 307 to 410, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:14.654323 8373 ts_tablet_manager.cc:902] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Initiating tablet copy from peer 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:14.654815 8373 tablet_copy_client.cc:308] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:43517
I20240501 19:57:14.665357 5150 tablet_copy_service.cc:140] P 43061d2e1908476eaf2b3bdc9a402ee2: Received BeginTabletCopySession request for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 from peer eaa8d49089fc428ab6e6601637abe221 ({username='slave'} at 127.30.1.70:55749)
I20240501 19:57:14.665468 5150 tablet_copy_service.cc:161] P 43061d2e1908476eaf2b3bdc9a402ee2: Beginning new tablet copy session on tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 from peer eaa8d49089fc428ab6e6601637abe221 at {username='slave'} at 127.30.1.70:55749: session id = eaa8d49089fc428ab6e6601637abe221-b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:14.666253 5150 tablet_copy_source_session.cc:205] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:14.667433 8373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.673118 8373 tablet_copy_client.cc:783] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:14.673517 8373 tablet_copy_client.cc:647] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:14.675218 8373 tablet_copy_client.cc:523] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:14.677604 8373 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:57:14.682263 8383 ts_tablet_manager.cc:902] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:14.683620 8383 tablet_copy_client.cc:308] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:14.684031 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet cfb7ca52d5ca4789888fe71d8b29b65f from peer 4f5fc46abef54f898b8eccf16432f798 ({username='slave'} at 127.30.1.69:51389)
I20240501 19:57:14.684113 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet cfb7ca52d5ca4789888fe71d8b29b65f from peer 4f5fc46abef54f898b8eccf16432f798 at {username='slave'} at 127.30.1.69:51389: session id = 4f5fc46abef54f898b8eccf16432f798-cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:14.684964 5281 tablet_copy_source_session.cc:205] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:14.685808 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb7ca52d5ca4789888fe71d8b29b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.691339 8383 tablet_copy_client.cc:783] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:14.691521 8383 tablet_copy_client.cc:647] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:14.700161 8383 tablet_copy_client.cc:523] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:14.701414 8383 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Bootstrap starting.
I20240501 19:57:14.725018 8395 ts_tablet_manager.cc:902] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:14.735237 8395 tablet_copy_client.cc:308] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:14.735777 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet fe3059729d0d4d7ebf4c2f39071919f6 from peer 4b2c3262271245a4b1acf221c8e5110f ({username='slave'} at 127.30.1.71:56179)
I20240501 19:57:14.735870 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet fe3059729d0d4d7ebf4c2f39071919f6 from peer 4b2c3262271245a4b1acf221c8e5110f at {username='slave'} at 127.30.1.71:56179: session id = 4b2c3262271245a4b1acf221c8e5110f-fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:14.736771 5019 tablet_copy_source_session.cc:205] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:14.745765 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3059729d0d4d7ebf4c2f39071919f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.756704 8395 tablet_copy_client.cc:783] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:14.757287 8395 tablet_copy_client.cc:647] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:14.791975 8395 tablet_copy_client.cc:523] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:14.793550 8395 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap starting.
I20240501 19:57:14.832703 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:14.837152 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:14.846683 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:14.857113 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:14.865487 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:14.870775 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:14.893007 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:14.909924 8427 ts_tablet_manager.cc:902] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:14.910454 8427 tablet_copy_client.cc:308] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:14.911007 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet a3fd829e098745598ccb473e5a0503b0 from peer 6489e19b12b44216824eeb036c208ae5 ({username='slave'} at 127.30.1.68:47723)
I20240501 19:57:14.911090 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet a3fd829e098745598ccb473e5a0503b0 from peer 6489e19b12b44216824eeb036c208ae5 at {username='slave'} at 127.30.1.68:47723: session id = 6489e19b12b44216824eeb036c208ae5-a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:14.911967 5019 tablet_copy_source_session.cc:205] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:14.921255 8427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3fd829e098745598ccb473e5a0503b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:14.923166 8427 tablet_copy_client.cc:783] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:14.923343 8427 tablet_copy_client.cc:647] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:14.928980 8427 tablet_copy_client.cc:523] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:14.930867 8427 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Bootstrap starting.
I20240501 19:57:15.020545 8433 ts_tablet_manager.cc:902] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Initiating tablet copy from peer 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361)
I20240501 19:57:15.021235 8433 tablet_copy_client.cc:308] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:46361
I20240501 19:57:15.023275 5281 tablet_copy_service.cc:140] P 4f272a2b03c244d69ff5595d457853ac: Received BeginTabletCopySession request for tablet fc16a5fd751d4f109b420ed953abe053 from peer eaa8d49089fc428ab6e6601637abe221 ({username='slave'} at 127.30.1.70:59581)
I20240501 19:57:15.023371 5281 tablet_copy_service.cc:161] P 4f272a2b03c244d69ff5595d457853ac: Beginning new tablet copy session on tablet fc16a5fd751d4f109b420ed953abe053 from peer eaa8d49089fc428ab6e6601637abe221 at {username='slave'} at 127.30.1.70:59581: session id = eaa8d49089fc428ab6e6601637abe221-fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:15.024269 5281 tablet_copy_source_session.cc:205] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:15.026468 8433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc16a5fd751d4f109b420ed953abe053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:15.031672 8433 tablet_copy_client.cc:783] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:15.031878 8433 tablet_copy_client.cc:647] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:15.039971 8433 tablet_copy_client.cc:523] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:15.041251 8433 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Bootstrap starting.
I20240501 19:57:15.221741 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:15.224200 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.224761 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.224956 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:15.229661 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:15.248015 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.326484 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:15.430559 8373 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Bootstrap replayed 1/1 log segments. Stats: ops{read=384 overwritten=0 applied=383 ignored=0} inserts{seen=1012 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:15.432281 8373 tablet_bootstrap.cc:492] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Bootstrap complete.
I20240501 19:57:15.433863 8373 ts_tablet_manager.cc:1367] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.756s user 0.270s sys 0.016s
I20240501 19:57:15.437328 8373 raft_consensus.cc:348] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:15.446838 8373 raft_consensus.cc:729] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.447574 8373 consensus_queue.cc:260] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 383, Last appended: 1.384, Last appended by leader: 384, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:15.448137 8373 ts_tablet_manager.cc:1398] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:57:15.450095 5150 tablet_copy_service.cc:342] P 43061d2e1908476eaf2b3bdc9a402ee2: Request end of tablet copy session eaa8d49089fc428ab6e6601637abe221-b6b4cf93c1be45218a4c0bc0f7cb4c97 received from {username='slave'} at 127.30.1.70:55749
I20240501 19:57:15.450167 5150 tablet_copy_service.cc:433] P 43061d2e1908476eaf2b3bdc9a402ee2: ending tablet copy session eaa8d49089fc428ab6e6601637abe221-b6b4cf93c1be45218a4c0bc0f7cb4c97 on tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 with peer eaa8d49089fc428ab6e6601637abe221
I20240501 19:57:15.454169 8395 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap replayed 1/1 log segments. Stats: ops{read=337 overwritten=0 applied=337 ignored=0} inserts{seen=966 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.456866 8395 tablet_bootstrap.cc:492] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Bootstrap complete.
I20240501 19:57:15.457111 8395 ts_tablet_manager.cc:1367] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Time spent bootstrapping tablet: real 0.664s user 0.252s sys 0.015s
I20240501 19:57:15.458189 8395 raft_consensus.cc:348] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:15.463111 8395 raft_consensus.cc:729] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2c3262271245a4b1acf221c8e5110f, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.463306 8395 consensus_queue.cc:260] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 337, Last appended: 1.337, Last appended by leader: 337, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:15.463660 8395 ts_tablet_manager.cc:1398] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:15.473726 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 4b2c3262271245a4b1acf221c8e5110f-fe3059729d0d4d7ebf4c2f39071919f6 received from {username='slave'} at 127.30.1.71:56179
I20240501 19:57:15.473829 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 4b2c3262271245a4b1acf221c8e5110f-fe3059729d0d4d7ebf4c2f39071919f6 on tablet fe3059729d0d4d7ebf4c2f39071919f6 with peer 4b2c3262271245a4b1acf221c8e5110f
I20240501 19:57:15.477517 8383 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Bootstrap replayed 1/1 log segments. Stats: ops{read=401 overwritten=0 applied=401 ignored=0} inserts{seen=1032 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.477859 8383 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Bootstrap complete.
I20240501 19:57:15.478106 8383 ts_tablet_manager.cc:1367] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Time spent bootstrapping tablet: real 0.777s user 0.292s sys 0.004s
I20240501 19:57:15.478543 8383 raft_consensus.cc:348] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:15.478709 8383 raft_consensus.cc:729] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fc46abef54f898b8eccf16432f798, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.478883 8383 consensus_queue.cc:260] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 401, Last appended: 1.401, Last appended by leader: 401, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:15.479207 8383 ts_tablet_manager.cc:1398] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:15.480794 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session 4f5fc46abef54f898b8eccf16432f798-cfb7ca52d5ca4789888fe71d8b29b65f received from {username='slave'} at 127.30.1.69:51389
I20240501 19:57:15.480868 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session 4f5fc46abef54f898b8eccf16432f798-cfb7ca52d5ca4789888fe71d8b29b65f on tablet cfb7ca52d5ca4789888fe71d8b29b65f with peer 4f5fc46abef54f898b8eccf16432f798
I20240501 19:57:15.578801 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:15.579820 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.592064 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:15.605633 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.622042 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:15.640487 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:15.705459 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:15.716693 8427 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=446 overwritten=0 applied=446 ignored=0} inserts{seen=1116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.717125 8427 tablet_bootstrap.cc:492] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Bootstrap complete.
I20240501 19:57:15.717425 8427 ts_tablet_manager.cc:1367] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Time spent bootstrapping tablet: real 0.787s user 0.303s sys 0.028s
I20240501 19:57:15.717880 8427 raft_consensus.cc:348] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:15.718056 8427 raft_consensus.cc:729] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.718209 8427 consensus_queue.cc:260] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 446, Last appended: 1.446, Last appended by leader: 446, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:15.718519 8427 ts_tablet_manager.cc:1398] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:15.719056 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 6489e19b12b44216824eeb036c208ae5-a3fd829e098745598ccb473e5a0503b0 received from {username='slave'} at 127.30.1.68:47723
I20240501 19:57:15.719128 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 6489e19b12b44216824eeb036c208ae5-a3fd829e098745598ccb473e5a0503b0 on tablet a3fd829e098745598ccb473e5a0503b0 with peer 6489e19b12b44216824eeb036c208ae5
I20240501 19:57:15.722604 6264 raft_consensus.cc:1203] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.424->[1.425-1.461] Dedup: 1.446->[1.447-1.461]
I20240501 19:57:15.772677 6397 raft_consensus.cc:1203] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.383->[1.384-1.422] Dedup: 1.401->[1.402-1.422]
I20240501 19:57:15.791052 8433 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Bootstrap replayed 1/1 log segments. Stats: ops{read=422 overwritten=0 applied=422 ignored=0} inserts{seen=1051 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:15.791471 8433 tablet_bootstrap.cc:492] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Bootstrap complete.
I20240501 19:57:15.791765 8433 ts_tablet_manager.cc:1367] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Time spent bootstrapping tablet: real 0.751s user 0.304s sys 0.017s
I20240501 19:57:15.794040 8433 raft_consensus.cc:348] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:15.794211 8433 raft_consensus.cc:729] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Initialized, Role: FOLLOWER
I20240501 19:57:15.794373 8433 consensus_queue.cc:260] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 422, Last appended: 1.422, Last appended by leader: 422, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:15.794823 8433 ts_tablet_manager.cc:1398] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:15.796947 5281 tablet_copy_service.cc:342] P 4f272a2b03c244d69ff5595d457853ac: Request end of tablet copy session eaa8d49089fc428ab6e6601637abe221-fc16a5fd751d4f109b420ed953abe053 received from {username='slave'} at 127.30.1.70:59581
I20240501 19:57:15.797019 5281 tablet_copy_service.cc:433] P 4f272a2b03c244d69ff5595d457853ac: ending tablet copy session eaa8d49089fc428ab6e6601637abe221-fc16a5fd751d4f109b420ed953abe053 on tablet fc16a5fd751d4f109b420ed953abe053 with peer eaa8d49089fc428ab6e6601637abe221
I20240501 19:57:15.838686 6670 raft_consensus.cc:1203] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.321->[1.322-1.357] Dedup: 1.337->[1.338-1.357]
I20240501 19:57:15.856225 6531 raft_consensus.cc:1203] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.354->[1.355-1.416] Dedup: 1.384->[1.385-1.416]
I20240501 19:57:15.916610 6530 raft_consensus.cc:1203] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.409->[1.410-1.442] Dedup: 1.422->[1.423-1.442]
I20240501 19:57:16.041834 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.046564 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:16.046749 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.059008 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:16.060082 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:16.066782 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:16.084137 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.380542 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.381163 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.402066 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:16.409236 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:16.415827 4977 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:16.422869 6246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:16.432124 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.546789 4996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6"
dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48924
I20240501 19:57:16.547042 4996 raft_consensus.cc:595] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:16.552064 7104 raft_consensus.cc:979] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: : Instructing follower 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:16.552199 7104 raft_consensus.cc:1067] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Signalling peer 6489e19b12b44216824eeb036c208ae5 to start an election
W20240501 19:57:16.559851 4959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37976: Service unavailable: leader transfer in progress
W20240501 19:57:16.572836 4959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37976: Service unavailable: leader transfer in progress
I20240501 19:57:16.574419 6265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6"
dest_uuid: "6489e19b12b44216824eeb036c208ae5"
from {username='slave'} at 127.30.1.65:51101
I20240501 19:57:16.574522 6265 raft_consensus.cc:482] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:16.574571 6265 raft_consensus.cc:3037] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:16.575618 6265 raft_consensus.cc:504] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:16.576093 6265 leader_election.cc:288] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835), 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049)
I20240501 19:57:16.576422 4996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 382 } } ignore_live_leader: true dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:16.576529 4996 raft_consensus.cc:3032] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:16.576575 4996 raft_consensus.cc:729] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:16.576732 4996 consensus_queue.cc:260] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 382, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:16.576897 4996 raft_consensus.cc:3037] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:16.577932 4996 raft_consensus.cc:2454] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:16.578442 6202 leader_election.cc:302] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6489e19b12b44216824eeb036c208ae5, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:16.578753 8598 raft_consensus.cc:2781] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:16.578910 8598 raft_consensus.cc:686] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Running, Role: LEADER
I20240501 19:57:16.579033 8598 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 382, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:16.580515 4849 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: term changed from 1 to 2, leader changed from b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) to 6489e19b12b44216824eeb036c208ae5 (127.30.1.68). New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
W20240501 19:57:16.581347 4959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37976: Illegal state: replica b051b1f42cb443e58813b92c5ae4a62e is not leader of this config: current role FOLLOWER
W20240501 19:57:16.597668 4959 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37976: Illegal state: replica b051b1f42cb443e58813b92c5ae4a62e is not leader of this config: current role FOLLOWER
I20240501 19:57:16.601785 6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3059729d0d4d7ebf4c2f39071919f6" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 382 } } ignore_live_leader: true dest_uuid: "4b2c3262271245a4b1acf221c8e5110f"
I20240501 19:57:16.601958 6671 raft_consensus.cc:3037] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:16.602910 6671 raft_consensus.cc:2454] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:16.704768 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.704823 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:16.725128 6671 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20240501 19:57:16.725345 4996 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20240501 19:57:16.725531 8598 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20240501 19:57:16.725668 8630 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20240501 19:57:16.727641 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:16.729089 8434 mvcc.cc:205] Tried to move back new op lower bound from 7022974716822675456 to 7022974716228673536. Current Snapshot: MvccSnapshot[applied={T|T < 7022974716822675456 or (T in {7022974716822675456})}]
I20240501 19:57:16.737314 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:16.764705 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:16.818200 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:16.836830 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.205752 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:17.208128 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.209393 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:17.210425 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.211066 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.219938 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:17.240523 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:17.350793 5260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f"
dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53016
I20240501 19:57:17.350937 5260 raft_consensus.cc:595] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Received request to transfer leadership
W20240501 19:57:17.371054 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.377820 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.384195 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.392942 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.406041 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.415298 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.422981 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.434957 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.444375 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.457211 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.469244 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.483374 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.496891 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.512084 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.530732 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
I20240501 19:57:17.547876 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:17.548648 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:57:17.551136 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
I20240501 19:57:17.561462 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:17.565349 5241 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:17.569187 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
I20240501 19:57:17.576823 6378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.584671 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:17.591970 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
I20240501 19:57:17.594695 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:57:17.614981 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.637814 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.660044 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.685576 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.709708 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.734798 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.762521 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.794183 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
W20240501 19:57:17.821456 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Service unavailable: leader transfer in progress
I20240501 19:57:17.823520 6586 raft_consensus.cc:979] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: : Instructing follower 4f5fc46abef54f898b8eccf16432f798 to start an election
I20240501 19:57:17.823598 6586 raft_consensus.cc:1067] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Signalling peer 4f5fc46abef54f898b8eccf16432f798 to start an election
I20240501 19:57:17.823957 6396 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f"
dest_uuid: "4f5fc46abef54f898b8eccf16432f798"
from {username='slave'} at 127.30.1.67:54585
I20240501 19:57:17.824136 6396 raft_consensus.cc:482] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:17.824257 6396 raft_consensus.cc:3037] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:17.825336 6396 raft_consensus.cc:504] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:17.826423 6396 leader_election.cc:288] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [CANDIDATE]: Term 2 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361), 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517)
I20240501 19:57:17.826761 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f" candidate_uuid: "4f5fc46abef54f898b8eccf16432f798" candidate_term: 2 candidate_status { last_received { term: 1 index: 468 } } ignore_live_leader: true dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:17.826874 5260 raft_consensus.cc:3032] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:17.826917 5260 raft_consensus.cc:729] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:17.826999 5127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb7ca52d5ca4789888fe71d8b29b65f" candidate_uuid: "4f5fc46abef54f898b8eccf16432f798" candidate_term: 2 candidate_status { last_received { term: 1 index: 468 } } ignore_live_leader: true dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:17.827039 5260 consensus_queue.cc:260] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 468, Committed index: 468, Last appended: 1.468, Last appended by leader: 468, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:17.827123 5127 raft_consensus.cc:3037] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:17.827172 5260 raft_consensus.cc:3037] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:17.828202 5260 raft_consensus.cc:2454] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5fc46abef54f898b8eccf16432f798 in term 2.
I20240501 19:57:17.828205 5127 raft_consensus.cc:2454] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5fc46abef54f898b8eccf16432f798 in term 2.
I20240501 19:57:17.828733 6333 leader_election.cc:302] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f272a2b03c244d69ff5595d457853ac, 4f5fc46abef54f898b8eccf16432f798; no voters:
I20240501 19:57:17.838073 8789 raft_consensus.cc:2781] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:17.838279 8789 raft_consensus.cc:686] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 LEADER]: Becoming Leader. State: Replica: 4f5fc46abef54f898b8eccf16432f798, State: Running, Role: LEADER
I20240501 19:57:17.838609 8789 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 468, Committed index: 468, Last appended: 1.468, Last appended by leader: 468, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:17.845865 4854 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 reported cstate change: term changed from 1 to 2, leader changed from 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) to 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69). New cstate: current_term: 2 leader_uuid: "4f5fc46abef54f898b8eccf16432f798" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
W20240501 19:57:17.853374 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Illegal state: replica 4f272a2b03c244d69ff5595d457853ac is not leader of this config: current role FOLLOWER
I20240501 19:57:17.890164 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:17.897087 5241 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:57:17.905565 5088 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35796: Illegal state: replica 43061d2e1908476eaf2b3bdc9a402ee2 is not leader of this config: current role FOLLOWER
I20240501 19:57:17.906037 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:17.907649 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:17.928694 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:17.939327 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 19:57:17.940071 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56386: Illegal state: replica 4f272a2b03c244d69ff5595d457853ac is not leader of this config: current role FOLLOWER
I20240501 19:57:17.957384 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:57:17.973100 5088 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35796: Illegal state: replica 43061d2e1908476eaf2b3bdc9a402ee2 is not leader of this config: current role FOLLOWER
I20240501 19:57:18.014009 5126 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Refusing update from remote peer 4f5fc46abef54f898b8eccf16432f798: Log matching property violated. Preceding OpId in replica: term: 1 index: 468. Preceding OpId from leader: term: 2 index: 470. (index mismatch)
I20240501 19:57:18.014025 5261 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Refusing update from remote peer 4f5fc46abef54f898b8eccf16432f798: Log matching property violated. Preceding OpId in replica: term: 1 index: 468. Preceding OpId from leader: term: 2 index: 470. (index mismatch)
I20240501 19:57:18.014413 8789 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 469, Last known committed idx: 468, Time since last communication: 0.000s
I20240501 19:57:18.015775 8789 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 469, Last known committed idx: 468, Time since last communication: 0.000s
I20240501 19:57:18.020448 8506 mvcc.cc:205] Tried to move back new op lower bound from 7022974722099884032 to 7022974721410805760. Current Snapshot: MvccSnapshot[applied={T|T < 7022974722099884032 or (T in {7022974722099884032})}]
I20240501 19:57:18.072379 5126 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97"
dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35806
I20240501 19:57:18.072542 5126 raft_consensus.cc:595] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:57:18.105500 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.107587 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.112237 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.121649 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.129201 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.137347 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.144994 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.154909 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.167281 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.181989 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.196939 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.210563 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.226871 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.253865 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
I20240501 19:57:18.257441 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:18.258901 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:18.262347 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:18.263177 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:18.268088 5241 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:57:18.273383 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
I20240501 19:57:18.273551 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:18.285719 6511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:57:18.289774 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.308877 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.326948 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.350411 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.375345 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.400485 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
W20240501 19:57:18.426764 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Service unavailable: leader transfer in progress
I20240501 19:57:18.431579 7166 raft_consensus.cc:979] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: : Instructing follower eaa8d49089fc428ab6e6601637abe221 to start an election
I20240501 19:57:18.431666 7166 raft_consensus.cc:1067] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Signalling peer eaa8d49089fc428ab6e6601637abe221 to start an election
I20240501 19:57:18.432482 6529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97"
dest_uuid: "eaa8d49089fc428ab6e6601637abe221"
from {username='slave'} at 127.30.1.66:40639
I20240501 19:57:18.432641 6529 raft_consensus.cc:482] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:18.432688 6529 raft_consensus.cc:3037] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:18.433590 6529 raft_consensus.cc:504] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:18.434484 6529 leader_election.cc:288] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [CANDIDATE]: Term 2 election: Requested vote from peers 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517), 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827)
I20240501 19:57:18.434707 5127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97" candidate_uuid: "eaa8d49089fc428ab6e6601637abe221" candidate_term: 2 candidate_status { last_received { term: 1 index: 496 } } ignore_live_leader: true dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
I20240501 19:57:18.434944 5127 raft_consensus.cc:3032] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:18.435034 5127 raft_consensus.cc:729] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:57:18.435295 5127 consensus_queue.cc:260] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 1.496, Last appended by leader: 496, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:18.435530 5127 raft_consensus.cc:3037] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:18.436527 5127 raft_consensus.cc:2454] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa8d49089fc428ab6e6601637abe221 in term 2.
I20240501 19:57:18.437165 6465 leader_election.cc:302] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, eaa8d49089fc428ab6e6601637abe221; no voters:
I20240501 19:57:18.437513 8853 raft_consensus.cc:2781] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:18.437690 8853 raft_consensus.cc:686] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 2 LEADER]: Becoming Leader. State: Replica: eaa8d49089fc428ab6e6601637abe221, State: Running, Role: LEADER
I20240501 19:57:18.438047 8853 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 1.496, Last appended by leader: 496, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:18.442925 4854 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 reported cstate change: term changed from 1 to 2, leader changed from 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) to eaa8d49089fc428ab6e6601637abe221 (127.30.1.70). New cstate: current_term: 2 leader_uuid: "eaa8d49089fc428ab6e6601637abe221" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
W20240501 19:57:18.451802 5090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39858: Illegal state: replica 43061d2e1908476eaf2b3bdc9a402ee2 is not leader of this config: current role FOLLOWER
I20240501 19:57:18.451961 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b4cf93c1be45218a4c0bc0f7cb4c97" candidate_uuid: "eaa8d49089fc428ab6e6601637abe221" candidate_term: 2 candidate_status { last_received { term: 1 index: 496 } } ignore_live_leader: true dest_uuid: "4f5fc46abef54f898b8eccf16432f798"
I20240501 19:57:18.452127 6397 raft_consensus.cc:3037] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:18.453182 6397 raft_consensus.cc:2454] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaa8d49089fc428ab6e6601637abe221 in term 2.
I20240501 19:57:18.547616 6396 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Refusing update from remote peer eaa8d49089fc428ab6e6601637abe221: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 2 index: 498. (index mismatch)
I20240501 19:57:18.547801 5127 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Refusing update from remote peer eaa8d49089fc428ab6e6601637abe221: Log matching property violated. Preceding OpId in replica: term: 1 index: 496. Preceding OpId from leader: term: 2 index: 498. (index mismatch)
I20240501 19:57:18.548044 8853 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240501 19:57:18.548501 8853 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240501 19:57:18.550606 8858 mvcc.cc:205] Tried to move back new op lower bound from 7022974724286496768 to 7022974723855335424. Current Snapshot: MvccSnapshot[applied={T|T < 7022974724286496768}]
I20240501 19:57:18.776432 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:18.795939 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:18.797590 5241 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:18.811758 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:18.813558 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:18.825177 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:18.849426 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:18.922817 5261 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053"
dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53066
I20240501 19:57:18.922945 5261 raft_consensus.cc:595] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Received request to transfer leadership
W20240501 19:57:18.999219 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.009328 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.014042 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.020283 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.028539 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.039750 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.047415 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.056788 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.068179 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.081581 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.094012 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.108121 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.122722 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.133863 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 19:57:19.139073 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.149451 5241 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:19.155973 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:19.156570 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 19:57:19.158977 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.175727 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:19.177824 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.195727 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.210419 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 19:57:19.220073 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.220477 6246 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:57:19.244119 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.269281 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
W20240501 19:57:19.292790 5219 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Service unavailable: leader transfer in progress
I20240501 19:57:19.296131 6127 raft_consensus.cc:979] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: : Instructing follower 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:19.296211 6127 raft_consensus.cc:1067] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Signalling peer 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:19.296603 6265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053"
dest_uuid: "6489e19b12b44216824eeb036c208ae5"
from {username='slave'} at 127.30.1.67:52057
I20240501 19:57:19.296725 6265 raft_consensus.cc:482] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:19.296802 6265 raft_consensus.cc:3037] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.297891 6265 raft_consensus.cc:504] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:19.298261 6265 leader_election.cc:288] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Requested vote from peers 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361), eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385)
I20240501 19:57:19.298734 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 544 } } ignore_live_leader: true dest_uuid: "4f272a2b03c244d69ff5595d457853ac"
I20240501 19:57:19.298846 5260 raft_consensus.cc:3032] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:19.298889 5260 raft_consensus.cc:729] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4f272a2b03c244d69ff5595d457853ac, State: Running, Role: LEADER
I20240501 19:57:19.299011 5260 consensus_queue.cc:260] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 544, Committed index: 544, Last appended: 1.544, Last appended by leader: 544, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:19.299155 5260 raft_consensus.cc:3037] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.299614 6531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc16a5fd751d4f109b420ed953abe053" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 544 } } ignore_live_leader: true dest_uuid: "eaa8d49089fc428ab6e6601637abe221"
I20240501 19:57:19.299727 6531 raft_consensus.cc:3037] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.300647 6531 raft_consensus.cc:2454] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:19.300994 5260 raft_consensus.cc:2454] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:19.300997 6200 leader_election.cc:302] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6489e19b12b44216824eeb036c208ae5, eaa8d49089fc428ab6e6601637abe221; no voters:
I20240501 19:57:19.301183 8945 raft_consensus.cc:2781] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:19.301283 8945 raft_consensus.cc:686] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Running, Role: LEADER
I20240501 19:57:19.301424 8945 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 544, Committed index: 544, Last appended: 1.544, Last appended by leader: 544, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:19.304361 4850 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: term changed from 1 to 2, leader changed from 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) to 6489e19b12b44216824eeb036c208ae5 (127.30.1.68). New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
W20240501 19:57:19.319670 5220 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56400: Illegal state: replica 4f272a2b03c244d69ff5595d457853ac is not leader of this config: current role FOLLOWER
I20240501 19:57:19.388437 6529 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 544. Preceding OpId from leader: term: 2 index: 546. (index mismatch)
I20240501 19:57:19.388782 5261 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 544. Preceding OpId from leader: term: 2 index: 546. (index mismatch)
I20240501 19:57:19.388945 8945 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 545, Last known committed idx: 544, Time since last communication: 0.000s
I20240501 19:57:19.389238 8945 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f272a2b03c244d69ff5595d457853ac" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 545, Last known committed idx: 544, Time since last communication: 0.000s
I20240501 19:57:19.597986 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:19.605072 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:19.615445 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:19.621560 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:19.628774 4977 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:19.629048 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:19.634032 6246 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:19.752846 4998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0"
dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49000
I20240501 19:57:19.768069 4998 raft_consensus.cc:595] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:19.768508 7858 raft_consensus.cc:979] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: : Instructing follower 6489e19b12b44216824eeb036c208ae5 to start an election
I20240501 19:57:19.768596 7858 raft_consensus.cc:1067] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Signalling peer 6489e19b12b44216824eeb036c208ae5 to start an election
W20240501 19:57:19.770666 8915 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37960: Service unavailable: leader transfer in progress
W20240501 19:57:19.775475 4956 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37960: Service unavailable: leader transfer in progress
I20240501 19:57:19.775602 6264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0"
dest_uuid: "6489e19b12b44216824eeb036c208ae5"
from {username='slave'} at 127.30.1.65:51101
I20240501 19:57:19.775885 6264 raft_consensus.cc:482] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:19.776019 6264 raft_consensus.cc:3037] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:19.778883 4956 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37960: Service unavailable: leader transfer in progress
I20240501 19:57:19.781855 6264 raft_consensus.cc:504] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:19.782678 6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 615 } } ignore_live_leader: true dest_uuid: "4b2c3262271245a4b1acf221c8e5110f"
I20240501 19:57:19.782846 6671 raft_consensus.cc:3037] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:19.784152 4956 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37960: Service unavailable: leader transfer in progress
I20240501 19:57:19.789366 6671 raft_consensus.cc:2454] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:19.790580 4998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fd829e098745598ccb473e5a0503b0" candidate_uuid: "6489e19b12b44216824eeb036c208ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 615 } } ignore_live_leader: true dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:19.790689 4998 raft_consensus.cc:3032] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:19.790736 4998 raft_consensus.cc:729] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:19.790894 4998 consensus_queue.cc:260] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 615, Committed index: 615, Last appended: 1.615, Last appended by leader: 615, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:19.782335 6264 leader_election.cc:288] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835), 4b2c3262271245a4b1acf221c8e5110f (127.30.1.71:35049)
I20240501 19:57:19.791044 4998 raft_consensus.cc:3037] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:19.792069 4998 raft_consensus.cc:2454] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6489e19b12b44216824eeb036c208ae5 in term 2.
I20240501 19:57:19.792418 6202 leader_election.cc:302] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6489e19b12b44216824eeb036c208ae5, b051b1f42cb443e58813b92c5ae4a62e; no voters:
W20240501 19:57:19.793030 4956 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37960: Illegal state: replica b051b1f42cb443e58813b92c5ae4a62e is not leader of this config: current role FOLLOWER
I20240501 19:57:19.793087 8945 raft_consensus.cc:2781] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:19.793226 8945 raft_consensus.cc:686] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Becoming Leader. State: Replica: 6489e19b12b44216824eeb036c208ae5, State: Running, Role: LEADER
I20240501 19:57:19.793370 8945 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 615, Committed index: 615, Last appended: 1.615, Last appended by leader: 615, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:19.795118 4850 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: term changed from 1 to 2, leader changed from b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) to 6489e19b12b44216824eeb036c208ae5 (127.30.1.68). New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:19.831113 4996 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 615. Preceding OpId from leader: term: 2 index: 617. (index mismatch)
I20240501 19:57:19.833009 6669 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 615. Preceding OpId from leader: term: 2 index: 617. (index mismatch)
I20240501 19:57:19.833395 8945 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 616, Last known committed idx: 615, Time since last communication: 0.000s
I20240501 19:57:19.839282 8598 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 616, Last known committed idx: 615, Time since last communication: 0.000s
I20240501 19:57:19.852319 9016 mvcc.cc:205] Tried to move back new op lower bound from 7022974729544683520 to 7022974729395036160. Current Snapshot: MvccSnapshot[applied={T|T < 7022974729544683520}]
I20240501 19:57:19.980319 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:19.989857 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:19.997366 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:19.998493 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.000506 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:20.091106 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:20.106496 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.228536 6264 consensus_queue.cc:237] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 633, Committed index: 633, Last appended: 2.633, Last appended by leader: 615, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } }
I20240501 19:57:20.230993 6669 raft_consensus.cc:1261] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 633. Preceding OpId from leader: term: 2 index: 634. (index mismatch)
I20240501 19:57:20.231400 8669 consensus_queue.cc:1035] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 633, Time since last communication: 0.000s
I20240501 19:57:20.233247 9031 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.634: config changed from index 425 to 634, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:20.234205 6669 raft_consensus.cc:2932] T a3fd829e098745598ccb473e5a0503b0 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Committing config change with OpId 2.634: config changed from index 425 to 634, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:20.235884 4850 catalog_manager.cc:5484] T a3fd829e098745598ccb473e5a0503b0 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 425 to 634, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 634 OBSOLETE_local: false peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } }
I20240501 19:57:20.239425 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet a3fd829e098745598ccb473e5a0503b0 with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 634) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:20.239981 9063 tablet_replica.cc:330] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:20.240181 9063 raft_consensus.cc:2229] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:20.240394 9063 raft_consensus.cc:2258] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:20.241604 9063 ts_tablet_manager.cc:1875] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:20.249440 9063 ts_tablet_manager.cc:1888] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.633
I20240501 19:57:20.249579 9063 log.cc:1196] T a3fd829e098745598ccb473e5a0503b0 P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:20.250841 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet a3fd829e098745598ccb473e5a0503b0 (table rebalance_test_table_0 [id=a322394687f54ba583f06778d3760b8b]) successfully deleted
I20240501 19:57:20.448148 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:20.455031 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:20.458458 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.497135 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:20.497362 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:20.497439 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:20.529232 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.674486 6264 consensus_queue.cc:237] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 545, Committed index: 545, Last appended: 2.545, Last appended by leader: 382, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } }
I20240501 19:57:20.675973 6670 raft_consensus.cc:1261] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 545. Preceding OpId from leader: term: 2 index: 546. (index mismatch)
I20240501 19:57:20.676659 9031 consensus_queue.cc:1035] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 546, Last known committed idx: 545, Time since last communication: 0.000s
I20240501 19:57:20.680343 9068 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.546: config changed from index 322 to 546, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:20.682941 4848 catalog_manager.cc:5484] T fe3059729d0d4d7ebf4c2f39071919f6 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 322 to 546, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:20.686305 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3059729d0d4d7ebf4c2f39071919f6 with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 546) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:20.686477 9063 tablet_replica.cc:330] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:20.686594 9063 raft_consensus.cc:2229] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:20.686698 9063 raft_consensus.cc:2258] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:20.687774 9063 ts_tablet_manager.cc:1875] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:20.689944 9063 ts_tablet_manager.cc:1888] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.545
I20240501 19:57:20.690017 9063 log.cc:1196] T fe3059729d0d4d7ebf4c2f39071919f6 P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:20.690801 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet fe3059729d0d4d7ebf4c2f39071919f6 (table rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]) successfully deleted
I20240501 19:57:20.695350 6670 raft_consensus.cc:2932] T fe3059729d0d4d7ebf4c2f39071919f6 P 4b2c3262271245a4b1acf221c8e5110f [term 2 FOLLOWER]: Committing config change with OpId 2.546: config changed from index 322 to 546, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 546 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "4b2c3262271245a4b1acf221c8e5110f" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 35049 } } }
I20240501 19:57:20.882884 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.887200 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:20.889287 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:20.894131 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:20.912592 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:20.917290 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:21.006624 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:21.112941 6397 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 590, Committed index: 590, Last appended: 2.590, Last appended by leader: 468, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 591 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } }
I20240501 19:57:21.114290 5126 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Refusing update from remote peer 4f5fc46abef54f898b8eccf16432f798: Log matching property violated. Preceding OpId in replica: term: 2 index: 590. Preceding OpId from leader: term: 2 index: 591. (index mismatch)
I20240501 19:57:21.114712 9166 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 591, Last known committed idx: 590, Time since last communication: 0.000s
I20240501 19:57:21.117817 9166 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 LEADER]: Committing config change with OpId 2.591: config changed from index 384 to 591, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 591 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:21.120558 4856 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 reported cstate change: config changed from index 384 to 591, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "4f5fc46abef54f898b8eccf16432f798" committed_config { opid_index: 591 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:21.122432 5127 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Committing config change with OpId 2.591: config changed from index 384 to 591, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 591 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } }
I20240501 19:57:21.123814 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb7ca52d5ca4789888fe71d8b29b65f with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 591) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:21.124431 9183 tablet_replica.cc:330] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:21.124619 9183 raft_consensus.cc:2229] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:21.124728 9183 raft_consensus.cc:2258] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:21.125706 9183 ts_tablet_manager.cc:1875] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:21.129691 9183 ts_tablet_manager.cc:1888] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.590
I20240501 19:57:21.129813 9183 log.cc:1196] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:21.130617 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet cfb7ca52d5ca4789888fe71d8b29b65f (table rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]) successfully deleted
I20240501 19:57:21.319878 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:21.326773 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:21.330824 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:21.341919 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:21.349400 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:21.350347 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:21.355988 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:21.523922 6530 consensus_queue.cc:237] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 563, Committed index: 563, Last appended: 2.563, Last appended by leader: 496, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 564 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:21.525053 6397 raft_consensus.cc:1261] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Refusing update from remote peer eaa8d49089fc428ab6e6601637abe221: Log matching property violated. Preceding OpId in replica: term: 2 index: 563. Preceding OpId from leader: term: 2 index: 564. (index mismatch)
I20240501 19:57:21.525481 9145 consensus_queue.cc:1035] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 564, Last known committed idx: 563, Time since last communication: 0.000s
I20240501 19:57:21.530346 9145 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 [term 2 LEADER]: Committing config change with OpId 2.564: config changed from index 355 to 564, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 564 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:21.532856 4856 catalog_manager.cc:5484] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P eaa8d49089fc428ab6e6601637abe221 reported cstate change: config changed from index 355 to 564, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "eaa8d49089fc428ab6e6601637abe221" committed_config { opid_index: 564 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:21.533744 6396 raft_consensus.cc:2932] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 4f5fc46abef54f898b8eccf16432f798 [term 2 FOLLOWER]: Committing config change with OpId 2.564: config changed from index 355 to 564, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 564 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:21.537503 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 564) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:21.537874 9239 tablet_replica.cc:330] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:21.538033 9239 raft_consensus.cc:2229] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:21.538208 9239 raft_consensus.cc:2258] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:21.539624 9239 ts_tablet_manager.cc:1875] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:21.548864 9239 ts_tablet_manager.cc:1888] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.563
I20240501 19:57:21.549010 9239 log.cc:1196] T b6b4cf93c1be45218a4c0bc0f7cb4c97 P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:21.549687 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet b6b4cf93c1be45218a4c0bc0f7cb4c97 (table rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]) successfully deleted
I20240501 19:57:21.903851 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:21.925567 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:21.940963 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:21.947804 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:21.956177 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:21.961952 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:21.977051 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:22.095999 6264 consensus_queue.cc:237] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 609, Committed index: 609, Last appended: 2.609, Last appended by leader: 544, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } }
I20240501 19:57:22.097191 6531 raft_consensus.cc:1261] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 609. Preceding OpId from leader: term: 2 index: 610. (index mismatch)
I20240501 19:57:22.101291 9031 consensus_queue.cc:1035] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 610, Last known committed idx: 609, Time since last communication: 0.000s
I20240501 19:57:22.104367 9031 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.610: config changed from index 410 to 610, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:22.106859 4856 catalog_manager.cc:5484] T fc16a5fd751d4f109b420ed953abe053 P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 410 to 610, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:22.107565 6531 raft_consensus.cc:2932] T fc16a5fd751d4f109b420ed953abe053 P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Committing config change with OpId 2.610: config changed from index 410 to 610, VOTER 4f272a2b03c244d69ff5595d457853ac (127.30.1.67) evicted. New config: { opid_index: 610 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } }
I20240501 19:57:22.113189 5241 tablet_service.cc:1514] Processing DeleteTablet for tablet fc16a5fd751d4f109b420ed953abe053 with delete_type TABLET_DATA_TOMBSTONED (TS 4f272a2b03c244d69ff5595d457853ac not found in new config with opid_index 610) from {username='slave'} at 127.0.0.1:60580
I20240501 19:57:22.115625 9311 tablet_replica.cc:330] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: stopping tablet replica
I20240501 19:57:22.115804 9311 raft_consensus.cc:2229] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:22.115963 9311 raft_consensus.cc:2258] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:22.116688 9311 ts_tablet_manager.cc:1875] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:22.133327 9311 ts_tablet_manager.cc:1888] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.609
I20240501 19:57:22.133394 9311 log.cc:1196] T fc16a5fd751d4f109b420ed953abe053 P 4f272a2b03c244d69ff5595d457853ac: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:22.133999 4843 catalog_manager.cc:4830] TS 4f272a2b03c244d69ff5595d457853ac (127.30.1.67:46361): tablet fc16a5fd751d4f109b420ed953abe053 (table rebalance_test_table_4 [id=b6b5b30cba464cacb65b3781b033bbd7]) successfully deleted
I20240501 19:57:22.575798 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:22.584481 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:22.618297 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:22.627146 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:22.657035 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:22.665583 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:22.676812 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:22.998631 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:23.004308 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:23.020089 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:23.047581 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:23.053929 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:23.057487 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:23.081933 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:23.776242 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:23.785478 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:23.786374 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:23.796339 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:23.818543 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:23.875800 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:23.879140 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:24.142143 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:24.150287 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:24.150842 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:24.158788 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:24.164116 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:24.176872 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:24.195164 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:24.702811 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:24.704810 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:24.705631 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:24.710808 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:24.716725 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:24.718183 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:24.718351 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
c129cc94b2be4bc1bb1290501c6a5390 | 127.30.1.126:44833 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.30.1.84:39013 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
43061d2e1908476eaf2b3bdc9a402ee2 | 127.30.1.66:43517 | HEALTHY | <none> | 3 | 0
4b2c3262271245a4b1acf221c8e5110f | 127.30.1.71:35049 | HEALTHY | <none> | 0 | 0
4f272a2b03c244d69ff5595d457853ac | 127.30.1.67:46361 | HEALTHY | <none> | 3 | 0
4f5fc46abef54f898b8eccf16432f798 | 127.30.1.69:33827 | HEALTHY | <none> | 1 | 0
6489e19b12b44216824eeb036c208ae5 | 127.30.1.68:34679 | HEALTHY | <none> | 4 | 0
b051b1f42cb443e58813b92c5ae4a62e | 127.30.1.65:32835 | HEALTHY | <none> | 4 | 0
eaa8d49089fc428ab6e6601637abe221 | 127.30.1.70:41385 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.30.1.65 | experimental | 127.30.1.65:32835
local_ip_for_outbound_sockets | 127.30.1.66 | experimental | 127.30.1.66:43517
local_ip_for_outbound_sockets | 127.30.1.67 | experimental | 127.30.1.67:46361
local_ip_for_outbound_sockets | 127.30.1.68 | experimental | 127.30.1.68:34679
local_ip_for_outbound_sockets | 127.30.1.69 | experimental | 127.30.1.69:33827
local_ip_for_outbound_sockets | 127.30.1.70 | experimental | 127.30.1.70:41385
local_ip_for_outbound_sockets | 127.30.1.71 | experimental | 127.30.1.71:35049
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.1.65:32835
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.1.66:43517
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.1.67:46361
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.1.68:34679
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.1.69:33827
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.1.70:41385
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.30.1.71:35049
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.30.1.84:39013 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:57:24.752738 30725 log_verifier.cc:126] Checking tablet 0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:24.874521 30725 log_verifier.cc:177] Verified matching terms for 776 ops in tablet 0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:24.874785 30725 log_verifier.cc:126] Checking tablet 0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:24.996153 30725 log_verifier.cc:177] Verified matching terms for 742 ops in tablet 0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:24.996330 30725 log_verifier.cc:126] Checking tablet 1fdb072bde014a35989544648d90249b
I20240501 19:57:25.117086 30725 log_verifier.cc:177] Verified matching terms for 785 ops in tablet 1fdb072bde014a35989544648d90249b
I20240501 19:57:25.117326 30725 log_verifier.cc:126] Checking tablet 259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:25.223227 30725 log_verifier.cc:177] Verified matching terms for 654 ops in tablet 259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:25.223454 30725 log_verifier.cc:126] Checking tablet 9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:25.334877 30725 log_verifier.cc:177] Verified matching terms for 699 ops in tablet 9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:25.335067 30725 log_verifier.cc:126] Checking tablet a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:25.350450 30725 log_verifier.cc:177] Verified matching terms for 4 ops in tablet a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:25.350505 30725 log_verifier.cc:126] Checking tablet a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:25.477514 30725 log_verifier.cc:177] Verified matching terms for 800 ops in tablet a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:25.477738 30725 log_verifier.cc:126] Checking tablet a7191b815b8747a082c26c1099a65518
I20240501 19:57:25.580544 30725 log_verifier.cc:177] Verified matching terms for 661 ops in tablet a7191b815b8747a082c26c1099a65518
I20240501 19:57:25.580718 30725 log_verifier.cc:126] Checking tablet aeaad9f75c694c84a890194769b4b461
I20240501 19:57:25.685580 30725 log_verifier.cc:177] Verified matching terms for 646 ops in tablet aeaad9f75c694c84a890194769b4b461
I20240501 19:57:25.685784 30725 log_verifier.cc:126] Checking tablet b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:25.792752 30725 log_verifier.cc:177] Verified matching terms for 653 ops in tablet b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:25.792966 30725 log_verifier.cc:126] Checking tablet cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:25.905835 30725 log_verifier.cc:177] Verified matching terms for 747 ops in tablet cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:25.906037 30725 log_verifier.cc:126] Checking tablet d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:26.018996 30725 log_verifier.cc:177] Verified matching terms for 720 ops in tablet d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:26.019253 30725 log_verifier.cc:126] Checking tablet e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:26.124176 30725 log_verifier.cc:177] Verified matching terms for 677 ops in tablet e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:26.124382 30725 log_verifier.cc:126] Checking tablet ecbb6869820148cbb31738a0efc7c515
I20240501 19:57:26.246049 30725 log_verifier.cc:177] Verified matching terms for 758 ops in tablet ecbb6869820148cbb31738a0efc7c515
I20240501 19:57:26.246276 30725 log_verifier.cc:126] Checking tablet fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:26.356060 30725 log_verifier.cc:177] Verified matching terms for 711 ops in tablet fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:26.356274 30725 log_verifier.cc:126] Checking tablet fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:26.458427 30725 log_verifier.cc:177] Verified matching terms for 666 ops in tablet fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:26.460261 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.72:0
--local_ip_for_outbound_sockets=127.30.1.72
--webserver_interface=127.30.1.72
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--builtin_ntp_servers=127.30.1.84:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:57:26.561216 9572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:26.561379 9572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:26.561419 9572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:26.564474 9572 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:26.564530 9572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:26.564630 9572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.72
I20240501 19:57:26.567957 9572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.1.72
--webserver_port=0
--tserver_master_addrs=127.30.1.126:44833
--never_fsync=true
--heap_profile_path=/tmp/kudu.9572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.72
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:26.568603 9572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:26.569633 9572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:26.577096 9577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:26.577713 9580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:26.578318 9578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:26.579155 9572 server_base.cc:1034] running on GCE node
I20240501 19:57:26.579458 9572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:26.579936 9572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:26.581084 9572 hybrid_clock.cc:648] HybridClock initialized: now 1714593446581057 us; error 37 us; skew 500 ppm
I20240501 19:57:26.588385 9572 webserver.cc:460] Webserver started at http://127.30.1.72:40993/ using document root <none> and password file <none>
I20240501 19:57:26.588869 9572 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:26.588960 9572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:26.589146 9572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:26.590827 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5a06aee2d39349528d5fca43c6658ebc"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-htmr"
I20240501 19:57:26.591370 9572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5a06aee2d39349528d5fca43c6658ebc"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-htmr"
I20240501 19:57:26.594758 9572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:26.596686 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.597595 9572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:26.597693 9572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5a06aee2d39349528d5fca43c6658ebc"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-htmr"
I20240501 19:57:26.597771 9572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:26.635689 9572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:26.636189 9572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:26.636341 9572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:26.636763 9572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:26.637495 9572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:26.637542 9572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.637590 9572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:26.637624 9572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.646005 9572 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.72:36501
I20240501 19:57:26.646026 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.72:36501 every 8 connection(s)
I20240501 19:57:26.646857 9572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:57:26.654655 9700 heartbeater.cc:347] Connected to a master server at 127.30.1.126:44833
I20240501 19:57:26.654829 9700 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:26.655215 9700 heartbeater.cc:511] Master 127.30.1.126:44833 requested a full tablet report, sending...
I20240501 19:57:26.655920 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 9572
I20240501 19:57:26.656028 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:57:26.656128 4854 ts_manager.cc:194] Registered new tserver with Master: 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501)
I20240501 19:57:26.656632 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.72:55699
I20240501 19:57:26.830050 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:26.832540 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:26.832973 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:26.835150 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:26.836769 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:26.847282 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:26.852737 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:26.854502 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:26.937314 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:26.939272 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:26.942178 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:26.952745 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:26.957525 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:26.960229 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:26.960467 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:26.980494 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.015211 6395 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 747, Committed index: 747, Last appended: 2.747, Last appended by leader: 468, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.016459 5127 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Refusing update from remote peer 4f5fc46abef54f898b8eccf16432f798: Log matching property violated. Preceding OpId in replica: term: 2 index: 747. Preceding OpId from leader: term: 2 index: 748. (index mismatch)
I20240501 19:57:27.016855 9790 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 748, Last known committed idx: 747, Time since last communication: 0.000s
I20240501 19:57:27.018517 9789 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 LEADER]: Committing config change with OpId 2.748: config changed from index 591 to 748, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.018873 5127 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Committing config change with OpId 2.748: config changed from index 591 to 748, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.021059 4848 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 reported cstate change: config changed from index 591 to 748, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New cstate: current_term: 2 leader_uuid: "4f5fc46abef54f898b8eccf16432f798" committed_config { opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
W20240501 19:57:27.022071 6331 consensus_peers.cc:485] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 -> Peer 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501): Couldn't send request to peer 5a06aee2d39349528d5fca43c6658ebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfb7ca52d5ca4789888fe71d8b29b65f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:27.037403 4999 consensus_queue.cc:237] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 677, Committed index: 677, Last appended: 1.677, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.038623 5127 raft_consensus.cc:1261] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 19:57:27.038934 9829 consensus_queue.cc:1035] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 19:57:27.040431 9326 raft_consensus.cc:2932] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.678: config changed from index -1 to 678, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.040855 5127 raft_consensus.cc:2932] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index -1 to 678, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
W20240501 19:57:27.041638 4932 consensus_peers.cc:485] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501): Couldn't send request to peer 5a06aee2d39349528d5fca43c6658ebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3613e90b6a747cbb4e1bf8eb9107675. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:27.043227 4848 catalog_manager.cc:5484] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index -1 to 678, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.058609 6265 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.060072 6530 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:27.060461 9848 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:27.062116 9031 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.062686 6530 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.065250 4855 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 4 to 5, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
W20240501 19:57:27.065941 6200 consensus_peers.cc:485] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 -> Peer 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501): Couldn't send request to peer 5a06aee2d39349528d5fca43c6658ebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a054a3f0d0ca40d98979620a3bf39a1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:27.079080 4999 consensus_queue.cc:237] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 646, Committed index: 646, Last appended: 1.646, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.080144 6265 raft_consensus.cc:1261] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 646. Preceding OpId from leader: term: 1 index: 647. (index mismatch)
W20240501 19:57:27.080386 4932 consensus_peers.cc:485] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e -> Peer 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501): Couldn't send request to peer 5a06aee2d39349528d5fca43c6658ebc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaad9f75c694c84a890194769b4b461. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:27.080505 9829 consensus_queue.cc:1035] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 647, Last known committed idx: 646, Time since last communication: 0.000s
I20240501 19:57:27.081995 9326 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.647: config changed from index 163 to 647, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.082443 6265 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.647: config changed from index 163 to 647, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New config: { opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.084297 4848 catalog_manager.cc:5484] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 163 to 647, VOTER 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:27.135887 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.140153 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.145406 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.147557 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.148877 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.154161 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.161048 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.171952 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.247704 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.248293 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.250661 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.258193 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.261705 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.262583 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.263463 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.289156 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.358722 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.360061 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.360117 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.361835 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.377779 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.378127 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.380836 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.392930 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.464639 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.468892 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.470319 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.477030 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.477563 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.478093 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.482035 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.494143 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.564813 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.566155 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.572052 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.574723 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.575039 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.589134 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.591250 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.605059 10054 ts_tablet_manager.cc:902] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Initiating tablet copy from peer 4f5fc46abef54f898b8eccf16432f798 (127.30.1.69:33827)
I20240501 19:57:27.605773 10054 tablet_copy_client.cc:308] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.69:33827
I20240501 19:57:27.606114 10055 ts_tablet_manager.cc:902] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:27.606525 10055 tablet_copy_client.cc:308] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:27.610617 10066 ts_tablet_manager.cc:902] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Initiating tablet copy from peer b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835)
I20240501 19:57:27.611686 10066 tablet_copy_client.cc:308] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:32835
I20240501 19:57:27.611945 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.615478 5019 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet aeaad9f75c694c84a890194769b4b461 from peer 5a06aee2d39349528d5fca43c6658ebc ({username='slave'} at 127.30.1.72:46315)
I20240501 19:57:27.615479 5018 tablet_copy_service.cc:140] P b051b1f42cb443e58813b92c5ae4a62e: Received BeginTabletCopySession request for tablet e3613e90b6a747cbb4e1bf8eb9107675 from peer 5a06aee2d39349528d5fca43c6658ebc ({username='slave'} at 127.30.1.72:46315)
I20240501 19:57:27.615592 5019 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet aeaad9f75c694c84a890194769b4b461 from peer 5a06aee2d39349528d5fca43c6658ebc at {username='slave'} at 127.30.1.72:46315: session id = 5a06aee2d39349528d5fca43c6658ebc-aeaad9f75c694c84a890194769b4b461
I20240501 19:57:27.615687 5018 tablet_copy_service.cc:161] P b051b1f42cb443e58813b92c5ae4a62e: Beginning new tablet copy session on tablet e3613e90b6a747cbb4e1bf8eb9107675 from peer 5a06aee2d39349528d5fca43c6658ebc at {username='slave'} at 127.30.1.72:46315: session id = 5a06aee2d39349528d5fca43c6658ebc-e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:27.616453 5018 tablet_copy_source_session.cc:205] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:27.616453 5019 tablet_copy_source_session.cc:205] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:27.618422 6418 tablet_copy_service.cc:140] P 4f5fc46abef54f898b8eccf16432f798: Received BeginTabletCopySession request for tablet cfb7ca52d5ca4789888fe71d8b29b65f from peer 5a06aee2d39349528d5fca43c6658ebc ({username='slave'} at 127.30.1.72:40435)
I20240501 19:57:27.618551 6418 tablet_copy_service.cc:161] P 4f5fc46abef54f898b8eccf16432f798: Beginning new tablet copy session on tablet cfb7ca52d5ca4789888fe71d8b29b65f from peer 5a06aee2d39349528d5fca43c6658ebc at {username='slave'} at 127.30.1.72:40435: session id = 5a06aee2d39349528d5fca43c6658ebc-cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:27.619753 6418 tablet_copy_source_session.cc:205] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:27.619931 10066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3613e90b6a747cbb4e1bf8eb9107675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.620654 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb7ca52d5ca4789888fe71d8b29b65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.621145 10055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaad9f75c694c84a890194769b4b461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.625437 10054 tablet_copy_client.cc:783] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:27.625526 10066 tablet_copy_client.cc:783] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:27.625648 10054 tablet_copy_client.cc:647] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:27.625702 10055 tablet_copy_client.cc:783] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:27.625756 10066 tablet_copy_client.cc:647] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:27.625902 10055 tablet_copy_client.cc:647] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:27.628816 10055 tablet_copy_client.cc:523] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:27.630345 10054 tablet_copy_client.cc:523] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:27.630631 10055 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap starting.
I20240501 19:57:27.630689 10066 tablet_copy_client.cc:523] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:27.631565 10054 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap starting.
I20240501 19:57:27.631832 10066 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap starting.
I20240501 19:57:27.658078 9700 heartbeater.cc:503] Master 127.30.1.126:44833 was elected leader, sending a full tablet report...
I20240501 19:57:27.658766 10071 ts_tablet_manager.cc:902] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Initiating tablet copy from peer 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679)
I20240501 19:57:27.659160 10071 tablet_copy_client.cc:308] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34679
I20240501 19:57:27.664320 6286 tablet_copy_service.cc:140] P 6489e19b12b44216824eeb036c208ae5: Received BeginTabletCopySession request for tablet a054a3f0d0ca40d98979620a3bf39a1d from peer 5a06aee2d39349528d5fca43c6658ebc ({username='slave'} at 127.30.1.72:48191)
I20240501 19:57:27.664456 6286 tablet_copy_service.cc:161] P 6489e19b12b44216824eeb036c208ae5: Beginning new tablet copy session on tablet a054a3f0d0ca40d98979620a3bf39a1d from peer 5a06aee2d39349528d5fca43c6658ebc at {username='slave'} at 127.30.1.72:48191: session id = 5a06aee2d39349528d5fca43c6658ebc-a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:27.665716 6286 tablet_copy_source_session.cc:205] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:27.666618 10071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a054a3f0d0ca40d98979620a3bf39a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.668249 10071 tablet_copy_client.cc:783] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:27.668366 10071 tablet_copy_client.cc:647] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:27.669734 10071 tablet_copy_client.cc:523] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:27.670729 10071 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap starting.
I20240501 19:57:27.673245 10071 log.cc:824] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:27.676358 10071 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:27.676605 10071 tablet_bootstrap.cc:492] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap complete.
I20240501 19:57:27.676839 10071 ts_tablet_manager.cc:1367] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:57:27.679729 10071 raft_consensus.cc:348] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.680011 10071 raft_consensus.cc:729] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a06aee2d39349528d5fca43c6658ebc, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.680488 10071 consensus_queue.cc:260] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eaa8d49089fc428ab6e6601637abe221" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 41385 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:27.681385 10071 ts_tablet_manager.cc:1398] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:57:27.682039 6286 tablet_copy_service.cc:342] P 6489e19b12b44216824eeb036c208ae5: Request end of tablet copy session 5a06aee2d39349528d5fca43c6658ebc-a054a3f0d0ca40d98979620a3bf39a1d received from {username='slave'} at 127.30.1.72:48191
I20240501 19:57:27.682112 6286 tablet_copy_service.cc:433] P 6489e19b12b44216824eeb036c208ae5: ending tablet copy session 5a06aee2d39349528d5fca43c6658ebc-a054a3f0d0ca40d98979620a3bf39a1d on tablet a054a3f0d0ca40d98979620a3bf39a1d with peer 5a06aee2d39349528d5fca43c6658ebc
I20240501 19:57:27.936707 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.951341 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.957687 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:27.967767 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:27.971685 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.972674 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:27.975657 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:27.979949 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.048655 9654 raft_consensus.cc:1203] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240501 19:57:28.124327 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.124809 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.131659 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.138379 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.163055 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.168133 10055 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=647 overwritten=0 applied=647 ignored=0} inserts{seen=1317 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:28.172370 10055 tablet_bootstrap.cc:492] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap complete.
I20240501 19:57:28.172871 10055 ts_tablet_manager.cc:1367] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Time spent bootstrapping tablet: real 0.542s user 0.424s sys 0.023s
I20240501 19:57:28.173535 10055 raft_consensus.cc:348] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.173732 10055 raft_consensus.cc:729] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a06aee2d39349528d5fca43c6658ebc, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.173926 10055 consensus_queue.cc:260] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 647, Last appended: 1.647, Last appended by leader: 647, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 647 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.174304 10055 ts_tablet_manager.cc:1398] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.174834 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 5a06aee2d39349528d5fca43c6658ebc-aeaad9f75c694c84a890194769b4b461 received from {username='slave'} at 127.30.1.72:46315
I20240501 19:57:28.174906 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 5a06aee2d39349528d5fca43c6658ebc-aeaad9f75c694c84a890194769b4b461 on tablet aeaad9f75c694c84a890194769b4b461 with peer 5a06aee2d39349528d5fca43c6658ebc
I20240501 19:57:28.177548 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.177876 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.189574 10066 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=678 overwritten=0 applied=678 ignored=0} inserts{seen=1350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:28.189947 10066 tablet_bootstrap.cc:492] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap complete.
I20240501 19:57:28.190500 10066 ts_tablet_manager.cc:1367] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Time spent bootstrapping tablet: real 0.559s user 0.429s sys 0.025s
I20240501 19:57:28.191143 10066 raft_consensus.cc:348] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.191268 10066 raft_consensus.cc:729] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a06aee2d39349528d5fca43c6658ebc, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.191407 10066 consensus_queue.cc:260] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 678, Last appended: 1.678, Last appended by leader: 678, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.191725 10066 ts_tablet_manager.cc:1398] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.191812 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.192530 5019 tablet_copy_service.cc:342] P b051b1f42cb443e58813b92c5ae4a62e: Request end of tablet copy session 5a06aee2d39349528d5fca43c6658ebc-e3613e90b6a747cbb4e1bf8eb9107675 received from {username='slave'} at 127.30.1.72:46315
I20240501 19:57:28.192596 5019 tablet_copy_service.cc:433] P b051b1f42cb443e58813b92c5ae4a62e: ending tablet copy session 5a06aee2d39349528d5fca43c6658ebc-e3613e90b6a747cbb4e1bf8eb9107675 on tablet e3613e90b6a747cbb4e1bf8eb9107675 with peer 5a06aee2d39349528d5fca43c6658ebc
I20240501 19:57:28.213302 10054 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap replayed 1/1 log segments. Stats: ops{read=748 overwritten=0 applied=748 ignored=0} inserts{seen=1376 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:28.213568 10054 tablet_bootstrap.cc:492] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Bootstrap complete.
I20240501 19:57:28.213801 10054 ts_tablet_manager.cc:1367] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Time spent bootstrapping tablet: real 0.582s user 0.480s sys 0.031s
I20240501 19:57:28.214190 10054 raft_consensus.cc:348] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.214275 10054 raft_consensus.cc:729] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a06aee2d39349528d5fca43c6658ebc, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.214383 10054 consensus_queue.cc:260] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 748, Last appended: 2.748, Last appended by leader: 748, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 748 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.214650 10054 ts_tablet_manager.cc:1398] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:28.215314 6418 tablet_copy_service.cc:342] P 4f5fc46abef54f898b8eccf16432f798: Request end of tablet copy session 5a06aee2d39349528d5fca43c6658ebc-cfb7ca52d5ca4789888fe71d8b29b65f received from {username='slave'} at 127.30.1.72:40435
I20240501 19:57:28.215407 6418 tablet_copy_service.cc:433] P 4f5fc46abef54f898b8eccf16432f798: ending tablet copy session 5a06aee2d39349528d5fca43c6658ebc-cfb7ca52d5ca4789888fe71d8b29b65f on tablet cfb7ca52d5ca4789888fe71d8b29b65f with peer 5a06aee2d39349528d5fca43c6658ebc
I20240501 19:57:28.230248 6265 consensus_queue.cc:237] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.231281 9654 raft_consensus.cc:1261] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Refusing update from remote peer 6489e19b12b44216824eeb036c208ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:28.231660 9848 consensus_queue.cc:1035] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:28.233840 10177 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.234274 9654 raft_consensus.cc:2932] T a054a3f0d0ca40d98979620a3bf39a1d P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.235997 4855 catalog_manager.cc:5484] T a054a3f0d0ca40d98979620a3bf39a1d P 6489e19b12b44216824eeb036c208ae5 reported cstate change: config changed from index 5 to 6, VOTER eaa8d49089fc428ab6e6601637abe221 (127.30.1.70) evicted. New cstate: current_term: 2 leader_uuid: "6489e19b12b44216824eeb036c208ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6489e19b12b44216824eeb036c208ae5" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34679 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.242290 6511 tablet_service.cc:1514] Processing DeleteTablet for tablet a054a3f0d0ca40d98979620a3bf39a1d with delete_type TABLET_DATA_TOMBSTONED (TS eaa8d49089fc428ab6e6601637abe221 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48044
I20240501 19:57:28.242672 10191 tablet_replica.cc:330] stopping tablet replica
I20240501 19:57:28.242938 10191 raft_consensus.cc:2229] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.243247 10191 raft_consensus.cc:2258] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.244666 10191 ts_tablet_manager.cc:1875] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:28.246660 10191 ts_tablet_manager.cc:1888] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:28.246747 10191 log.cc:1196] T a054a3f0d0ca40d98979620a3bf39a1d P eaa8d49089fc428ab6e6601637abe221: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/wals/a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:28.247556 4840 catalog_manager.cc:4830] TS eaa8d49089fc428ab6e6601637abe221 (127.30.1.70:41385): tablet a054a3f0d0ca40d98979620a3bf39a1d (table TestTable [id=c74331f516e2486c896a4f5b5b485c80]) successfully deleted
I20240501 19:57:28.290156 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.296257 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.297091 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.297356 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.300690 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.309998 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.313057 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.314997 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.395673 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.399922 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.405545 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.406219 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.409164 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.411285 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.413868 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.439852 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.448940 9654 raft_consensus.cc:1203] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.646->[1.647-1.647] Dedup: 1.647->[]
I20240501 19:57:28.502313 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.505327 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.511539 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.511895 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.517138 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.524358 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.534624 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.540977 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.627990 9654 raft_consensus.cc:1203] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.747->[2.748-2.748] Dedup: 2.748->[]
I20240501 19:57:28.727429 9654 raft_consensus.cc:1203] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.677->[1.678-1.678] Dedup: 1.678->[]
I20240501 19:57:28.818389 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.822917 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.823992 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.829210 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.831665 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.832723 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.840088 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.859475 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.890271 4999 consensus_queue.cc:237] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 647, Committed index: 647, Last appended: 1.647, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 648 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:28.891263 9654 raft_consensus.cc:1261] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Refusing update from remote peer b051b1f42cb443e58813b92c5ae4a62e: Log matching property violated. Preceding OpId in replica: term: 1 index: 647. Preceding OpId from leader: term: 1 index: 648. (index mismatch)
I20240501 19:57:28.891625 9868 consensus_queue.cc:1035] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 648, Last known committed idx: 647, Time since last communication: 0.000s
I20240501 19:57:28.893205 10349 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Committing config change with OpId 1.648: config changed from index 647 to 648, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) evicted. New config: { opid_index: 648 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.893579 9654 raft_consensus.cc:2932] T aeaad9f75c694c84a890194769b4b461 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Committing config change with OpId 1.648: config changed from index 647 to 648, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) evicted. New config: { opid_index: 648 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.895720 4854 catalog_manager.cc:5484] T aeaad9f75c694c84a890194769b4b461 P b051b1f42cb443e58813b92c5ae4a62e reported cstate change: config changed from index 647 to 648, VOTER 6489e19b12b44216824eeb036c208ae5 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "b051b1f42cb443e58813b92c5ae4a62e" committed_config { opid_index: 648 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:28.903463 6246 tablet_service.cc:1514] Processing DeleteTablet for tablet aeaad9f75c694c84a890194769b4b461 with delete_type TABLET_DATA_TOMBSTONED (TS 6489e19b12b44216824eeb036c208ae5 not found in new config with opid_index 648) from {username='slave'} at 127.0.0.1:43120
I20240501 19:57:28.903913 10365 tablet_replica.cc:330] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: stopping tablet replica
I20240501 19:57:28.904197 10365 raft_consensus.cc:2229] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.904444 10365 raft_consensus.cc:2258] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.906183 10365 ts_tablet_manager.cc:1875] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:28.908015 10365 ts_tablet_manager.cc:1888] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.647
I20240501 19:57:28.908092 10365 log.cc:1196] T aeaad9f75c694c84a890194769b4b461 P 6489e19b12b44216824eeb036c208ae5: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/aeaad9f75c694c84a890194769b4b461
I20240501 19:57:28.908660 4842 catalog_manager.cc:4830] TS 6489e19b12b44216824eeb036c208ae5 (127.30.1.68:34679): tablet aeaad9f75c694c84a890194769b4b461 (table rebalance_test_table_2 [id=891c114dce494de2a4fca1a4a54abd90]) successfully deleted
I20240501 19:57:28.957211 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.958511 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.960134 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:28.960425 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:28.963586 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.978446 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:28.979183 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:28.986788 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.025260 6395 consensus_queue.cc:237] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 748, Committed index: 748, Last appended: 2.748, Last appended by leader: 468, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 749 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:29.026401 9654 raft_consensus.cc:1261] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Refusing update from remote peer 4f5fc46abef54f898b8eccf16432f798: Log matching property violated. Preceding OpId in replica: term: 2 index: 748. Preceding OpId from leader: term: 2 index: 749. (index mismatch)
I20240501 19:57:29.026769 9790 consensus_queue.cc:1035] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 749, Last known committed idx: 748, Time since last communication: 0.000s
I20240501 19:57:29.028561 10418 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 [term 2 LEADER]: Committing config change with OpId 2.749: config changed from index 748 to 749, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 749 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:29.028977 9654 raft_consensus.cc:2932] T cfb7ca52d5ca4789888fe71d8b29b65f P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Committing config change with OpId 2.749: config changed from index 748 to 749, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New config: { opid_index: 749 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:29.030972 4848 catalog_manager.cc:5484] T cfb7ca52d5ca4789888fe71d8b29b65f P 4f5fc46abef54f898b8eccf16432f798 reported cstate change: config changed from index 748 to 749, VOTER 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "4f5fc46abef54f898b8eccf16432f798" committed_config { opid_index: 749 OBSOLETE_local: false peers { permanent_uuid: "4f5fc46abef54f898b8eccf16432f798" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 33827 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:29.033665 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb7ca52d5ca4789888fe71d8b29b65f with delete_type TABLET_DATA_TOMBSTONED (TS 43061d2e1908476eaf2b3bdc9a402ee2 not found in new config with opid_index 749) from {username='slave'} at 127.0.0.1:48192
I20240501 19:57:29.033987 10424 tablet_replica.cc:330] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: stopping tablet replica
I20240501 19:57:29.034121 10424 raft_consensus.cc:2229] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:29.034243 10424 raft_consensus.cc:2258] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.035470 10424 ts_tablet_manager.cc:1875] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:29.037549 10424 ts_tablet_manager.cc:1888] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.748
I20240501 19:57:29.037678 10424 log.cc:1196] T cfb7ca52d5ca4789888fe71d8b29b65f P 43061d2e1908476eaf2b3bdc9a402ee2: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:29.038180 4842 catalog_manager.cc:4830] TS 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66:43517): tablet cfb7ca52d5ca4789888fe71d8b29b65f (table rebalance_test_table_1 [id=4dc6b2489e674228af5b095d4efdd26b]) successfully deleted
I20240501 19:57:29.082938 5108 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.083807 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.086474 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.089066 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.090785 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.106448 4977 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.107386 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.120142 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.141500 4999 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675"
dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57596
I20240501 19:57:29.141615 4999 raft_consensus.cc:595] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:29.150377 9868 raft_consensus.cc:979] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: : Instructing follower 43061d2e1908476eaf2b3bdc9a402ee2 to start an election
I20240501 19:57:29.150468 9868 raft_consensus.cc:1067] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Signalling peer 43061d2e1908476eaf2b3bdc9a402ee2 to start an election
I20240501 19:57:29.150880 5127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675"
dest_uuid: "43061d2e1908476eaf2b3bdc9a402ee2"
from {username='slave'} at 127.30.1.65:44573
I20240501 19:57:29.151018 5127 raft_consensus.cc:482] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:29.151085 5127 raft_consensus.cc:3037] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:29.151903 5127 raft_consensus.cc:504] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:29.152419 5127 leader_election.cc:288] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 2 election: Requested vote from peers b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835), 5a06aee2d39349528d5fca43c6658ebc (127.30.1.72:36501)
I20240501 19:57:29.152878 4999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 2 candidate_status { last_received { term: 1 index: 678 } } ignore_live_leader: true dest_uuid: "b051b1f42cb443e58813b92c5ae4a62e"
I20240501 19:57:29.153012 4999 raft_consensus.cc:3032] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:29.153055 4999 raft_consensus.cc:729] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b051b1f42cb443e58813b92c5ae4a62e, State: Running, Role: LEADER
I20240501 19:57:29.153203 4999 consensus_queue.cc:260] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 678, Committed index: 678, Last appended: 1.678, Last appended by leader: 678, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:29.153352 4999 raft_consensus.cc:3037] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:29.154049 4999 raft_consensus.cc:2454] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 2.
I20240501 19:57:29.154322 5066 leader_election.cc:302] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43061d2e1908476eaf2b3bdc9a402ee2, b051b1f42cb443e58813b92c5ae4a62e; no voters:
I20240501 19:57:29.154632 8233 raft_consensus.cc:2781] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:29.154780 8233 raft_consensus.cc:686] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 LEADER]: Becoming Leader. State: Replica: 43061d2e1908476eaf2b3bdc9a402ee2, State: Running, Role: LEADER
I20240501 19:57:29.154912 8233 consensus_queue.cc:237] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 678, Committed index: 678, Last appended: 1.678, Last appended by leader: 678, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:29.156745 4848 catalog_manager.cc:5484] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: term changed from 1 to 2, leader changed from b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) to 43061d2e1908476eaf2b3bdc9a402ee2 (127.30.1.66). New cstate: current_term: 2 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 } } peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:29.158447 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3613e90b6a747cbb4e1bf8eb9107675" candidate_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" candidate_term: 2 candidate_status { last_received { term: 1 index: 678 } } ignore_live_leader: true dest_uuid: "5a06aee2d39349528d5fca43c6658ebc"
I20240501 19:57:29.158600 9654 raft_consensus.cc:3037] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:29.159503 9654 raft_consensus.cc:2454] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43061d2e1908476eaf2b3bdc9a402ee2 in term 2.
I20240501 19:57:29.189159 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.191627 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.194814 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.200534 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.201453 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.206832 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.212466 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.212546 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.309370 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.309798 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.310580 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.312413 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.318629 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.320298 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.345077 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.345455 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.418996 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.419551 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.425439 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.429309 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.433437 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.445987 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.446072 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.468155 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.736572 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.738322 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.738509 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.745945 9654 raft_consensus.cc:1261] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 2 index: 679. (index mismatch)
I20240501 19:57:29.746435 8233 consensus_queue.cc:1035] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 19:57:29.748282 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:29.748708 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:29.748865 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:29.751843 4999 raft_consensus.cc:1261] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 2 index: 679. (index mismatch)
I20240501 19:57:29.752993 8233 consensus_queue.cc:1035] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b051b1f42cb443e58813b92c5ae4a62e" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 19:57:29.772177 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:29.783322 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.057662 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.057803 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.058292 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.059218 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.079177 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.079871 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.085232 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.089941 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.122360 5127 consensus_queue.cc:237] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 679, Committed index: 679, Last appended: 2.679, Last appended by leader: 678, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 680 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } }
I20240501 19:57:30.123247 9654 raft_consensus.cc:1261] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Refusing update from remote peer 43061d2e1908476eaf2b3bdc9a402ee2: Log matching property violated. Preceding OpId in replica: term: 2 index: 679. Preceding OpId from leader: term: 2 index: 680. (index mismatch)
I20240501 19:57:30.123564 10628 consensus_queue.cc:1035] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 680, Last known committed idx: 679, Time since last communication: 0.000s
I20240501 19:57:30.124763 8233 raft_consensus.cc:2932] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 [term 2 LEADER]: Committing config change with OpId 2.680: config changed from index 678 to 680, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 680 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:30.125111 9654 raft_consensus.cc:2932] T e3613e90b6a747cbb4e1bf8eb9107675 P 5a06aee2d39349528d5fca43c6658ebc [term 2 FOLLOWER]: Committing config change with OpId 2.680: config changed from index 678 to 680, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New config: { opid_index: 680 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:30.126839 4850 catalog_manager.cc:5484] T e3613e90b6a747cbb4e1bf8eb9107675 P 43061d2e1908476eaf2b3bdc9a402ee2 reported cstate change: config changed from index 678 to 680, VOTER b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" committed_config { opid_index: 680 OBSOLETE_local: false peers { permanent_uuid: "43061d2e1908476eaf2b3bdc9a402ee2" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 43517 } } peers { permanent_uuid: "5a06aee2d39349528d5fca43c6658ebc" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 36501 } } }
I20240501 19:57:30.129678 4977 tablet_service.cc:1514] Processing DeleteTablet for tablet e3613e90b6a747cbb4e1bf8eb9107675 with delete_type TABLET_DATA_TOMBSTONED (TS b051b1f42cb443e58813b92c5ae4a62e not found in new config with opid_index 680) from {username='slave'} at 127.0.0.1:55162
I20240501 19:57:30.129951 10691 tablet_replica.cc:330] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: stopping tablet replica
I20240501 19:57:30.130076 10691 raft_consensus.cc:2229] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.130193 10691 raft_consensus.cc:2258] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.131438 10691 ts_tablet_manager.cc:1875] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:30.133178 10691 ts_tablet_manager.cc:1888] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.679
I20240501 19:57:30.133244 10691 log.cc:1196] T e3613e90b6a747cbb4e1bf8eb9107675 P b051b1f42cb443e58813b92c5ae4a62e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:30.133765 4843 catalog_manager.cc:4830] TS b051b1f42cb443e58813b92c5ae4a62e (127.30.1.65:32835): tablet e3613e90b6a747cbb4e1bf8eb9107675 (table rebalance_test_table_3 [id=31aed2b22bb443479ceb35cde40f67f4]) successfully deleted
I20240501 19:57:30.384732 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.387337 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.390475 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.393537 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.399391 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.405371 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.413749 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.417554 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.498950 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.499034 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.510360 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.511049 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.511706 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.513034 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.521975 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.522346 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.626435 5108 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.629055 6378 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.633944 9634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.639084 5241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:30.639642 6511 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:30.650504 6246 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:30.657426 6651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:30.672102 4977 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
c129cc94b2be4bc1bb1290501c6a5390 | 127.30.1.126:44833 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.30.1.84:39013 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
43061d2e1908476eaf2b3bdc9a402ee2 | 127.30.1.66:43517 | HEALTHY | <none> | 4 | 0
4b2c3262271245a4b1acf221c8e5110f | 127.30.1.71:35049 | HEALTHY | <none> | 0 | 0
4f272a2b03c244d69ff5595d457853ac | 127.30.1.67:46361 | HEALTHY | <none> | 3 | 0
4f5fc46abef54f898b8eccf16432f798 | 127.30.1.69:33827 | HEALTHY | <none> | 1 | 0
5a06aee2d39349528d5fca43c6658ebc | 127.30.1.72:36501 | HEALTHY | <none> | 0 | 0
6489e19b12b44216824eeb036c208ae5 | 127.30.1.68:34679 | HEALTHY | <none> | 4 | 0
b051b1f42cb443e58813b92c5ae4a62e | 127.30.1.65:32835 | HEALTHY | <none> | 3 | 0
eaa8d49089fc428ab6e6601637abe221 | 127.30.1.70:41385 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.30.1.65 | experimental | 127.30.1.65:32835
local_ip_for_outbound_sockets | 127.30.1.66 | experimental | 127.30.1.66:43517
local_ip_for_outbound_sockets | 127.30.1.67 | experimental | 127.30.1.67:46361
local_ip_for_outbound_sockets | 127.30.1.68 | experimental | 127.30.1.68:34679
local_ip_for_outbound_sockets | 127.30.1.69 | experimental | 127.30.1.69:33827
local_ip_for_outbound_sockets | 127.30.1.70 | experimental | 127.30.1.70:41385
local_ip_for_outbound_sockets | 127.30.1.71 | experimental | 127.30.1.71:35049
local_ip_for_outbound_sockets | 127.30.1.72 | experimental | 127.30.1.72:36501
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.1.65:32835
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.1.66:43517
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.1.67:46361
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.30.1.68:34679
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.30.1.69:33827
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.30.1.70:41385
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.30.1.71:35049
server_dump_info_path | /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.30.1.72:36501
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.30.1.84:39013 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:57:30.691649 30725 log_verifier.cc:126] Checking tablet 0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:30.801623 30725 log_verifier.cc:177] Verified matching terms for 776 ops in tablet 0ca2013d6f6b496e99af4a248bd3ed57
I20240501 19:57:30.801880 30725 log_verifier.cc:126] Checking tablet 0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:30.909018 30725 log_verifier.cc:177] Verified matching terms for 742 ops in tablet 0cbdfb0387a9449f897a454ff26276b4
I20240501 19:57:30.909230 30725 log_verifier.cc:126] Checking tablet 1fdb072bde014a35989544648d90249b
I20240501 19:57:31.022478 30725 log_verifier.cc:177] Verified matching terms for 785 ops in tablet 1fdb072bde014a35989544648d90249b
I20240501 19:57:31.022765 30725 log_verifier.cc:126] Checking tablet 259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:31.121196 30725 log_verifier.cc:177] Verified matching terms for 654 ops in tablet 259f0c2339954fa7b9a1d4e604a8511a
I20240501 19:57:31.121366 30725 log_verifier.cc:126] Checking tablet 9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:31.223268 30725 log_verifier.cc:177] Verified matching terms for 699 ops in tablet 9b0753c139624897b37cab9f3a9eab35
I20240501 19:57:31.223448 30725 log_verifier.cc:126] Checking tablet a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:31.234977 30725 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a054a3f0d0ca40d98979620a3bf39a1d
I20240501 19:57:31.235030 30725 log_verifier.cc:126] Checking tablet a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:31.361434 30725 log_verifier.cc:177] Verified matching terms for 800 ops in tablet a3fd829e098745598ccb473e5a0503b0
I20240501 19:57:31.361640 30725 log_verifier.cc:126] Checking tablet a7191b815b8747a082c26c1099a65518
I20240501 19:57:31.456950 30725 log_verifier.cc:177] Verified matching terms for 661 ops in tablet a7191b815b8747a082c26c1099a65518
I20240501 19:57:31.457108 30725 log_verifier.cc:126] Checking tablet aeaad9f75c694c84a890194769b4b461
I20240501 19:57:31.553481 30725 log_verifier.cc:177] Verified matching terms for 648 ops in tablet aeaad9f75c694c84a890194769b4b461
I20240501 19:57:31.553675 30725 log_verifier.cc:126] Checking tablet b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:31.651489 30725 log_verifier.cc:177] Verified matching terms for 653 ops in tablet b6b4cf93c1be45218a4c0bc0f7cb4c97
I20240501 19:57:31.651685 30725 log_verifier.cc:126] Checking tablet cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:31.759617 30725 log_verifier.cc:177] Verified matching terms for 749 ops in tablet cfb7ca52d5ca4789888fe71d8b29b65f
I20240501 19:57:31.759832 30725 log_verifier.cc:126] Checking tablet d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:31.863318 30725 log_verifier.cc:177] Verified matching terms for 720 ops in tablet d1656d575d7f4d79bb289c3a043cf6aa
I20240501 19:57:31.863520 30725 log_verifier.cc:126] Checking tablet e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:31.964475 30725 log_verifier.cc:177] Verified matching terms for 680 ops in tablet e3613e90b6a747cbb4e1bf8eb9107675
I20240501 19:57:31.964707 30725 log_verifier.cc:126] Checking tablet ecbb6869820148cbb31738a0efc7c515
I20240501 19:57:32.076866 30725 log_verifier.cc:177] Verified matching terms for 758 ops in tablet ecbb6869820148cbb31738a0efc7c515
I20240501 19:57:32.077113 30725 log_verifier.cc:126] Checking tablet fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:32.178560 30725 log_verifier.cc:177] Verified matching terms for 711 ops in tablet fc16a5fd751d4f109b420ed953abe053
I20240501 19:57:32.178759 30725 log_verifier.cc:126] Checking tablet fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:32.277220 30725 log_verifier.cc:177] Verified matching terms for 666 ops in tablet fe3059729d0d4d7ebf4c2f39071919f6
I20240501 19:57:32.278764 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4916
I20240501 19:57:32.286727 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5048
I20240501 19:57:32.292654 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 5179
I20240501 19:57:32.298965 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 6184
I20240501 19:57:32.304591 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 6315
I20240501 19:57:32.310387 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 6447
I20240501 19:57:32.315871 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 6580
I20240501 19:57:32.320982 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 9572
I20240501 19:57:32.326159 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 4824
2024-05-01T19:57:32Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (38535 ms)
[----------] 1 test from RebalanceParamTest (38535 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240501 19:57:32.382515 30725 test_util.cc:268] Using random seed: -441853648
I20240501 19:57:32.383210 30725 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:57:32.383255 30725 ts_itest-base.cc:113] --------------
I20240501 19:57:32.383265 30725 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:57:32.383278 30725 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:57:32.383306 30725 ts_itest-base.cc:116] --------------
2024-05-01T19:57:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:32Z Disabled control of system clock
I20240501 19:57:32.390993 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:33217
--webserver_interface=127.30.1.126
--webserver_port=0
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.1.126:33217
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 19:57:32.487198 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:32.487357 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:32.487390 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:32.490123 10810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:32.490172 10810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:32.490195 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:32.490211 10810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:32.490226 10810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:32.493570 10810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.30.1.126:33217
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.1.126:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.1.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:32.494216 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:32.495111 10810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:32.501858 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.502496 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.503234 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:32.503990 10810 server_base.cc:1034] running on GCE node
I20240501 19:57:32.504385 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:32.505122 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:32.506286 10810 hybrid_clock.cc:648] HybridClock initialized: now 1714593452506271 us; error 32 us; skew 500 ppm
I20240501 19:57:32.513212 10810 webserver.cc:460] Webserver started at http://127.30.1.126:46249/ using document root <none> and password file <none>
I20240501 19:57:32.513667 10810 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:32.513720 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:32.513859 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:32.515206 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "06cf8280f20843729e99a32633f62d20"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.515635 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "06cf8280f20843729e99a32633f62d20"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.518679 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 19:57:32.520474 10824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.521375 10810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:32.521492 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
uuid: "06cf8280f20843729e99a32633f62d20"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.521572 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:32.532296 10810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:32.532785 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:32.532969 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:32.538697 10810 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.126:33217
I20240501 19:57:32.538722 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.126:33217 every 8 connection(s)
I20240501 19:57:32.539471 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:57:32.541621 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.545413 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 10810
I20240501 19:57:32.545543 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:57:32.545850 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: Bootstrap starting.
I20240501 19:57:32.547703 10877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.548314 10877 log.cc:824] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:32.549759 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: No bootstrap required, opened a new log
I20240501 19:57:32.552412 10877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } }
I20240501 19:57:32.552557 10877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.552590 10877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cf8280f20843729e99a32633f62d20, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.553018 10877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } }
I20240501 19:57:32.553155 10877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:32.553215 10877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:32.553313 10877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.554059 10877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } }
I20240501 19:57:32.554329 10877 leader_election.cc:302] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cf8280f20843729e99a32633f62d20; no voters:
I20240501 19:57:32.554510 10877 leader_election.cc:288] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:32.554817 10882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:32.555001 10882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [term 1 LEADER]: Becoming Leader. State: Replica: 06cf8280f20843729e99a32633f62d20, State: Running, Role: LEADER
I20240501 19:57:32.555258 10882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } }
I20240501 19:57:32.555326 10877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:32.556521 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cf8280f20843729e99a32633f62d20. Latest consensus state: current_term: 1 leader_uuid: "06cf8280f20843729e99a32633f62d20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } } }
I20240501 19:57:32.556808 10883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cf8280f20843729e99a32633f62d20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cf8280f20843729e99a32633f62d20" member_type: VOTER last_known_addr { host: "127.30.1.126" port: 33217 } } }
I20240501 19:57:32.556905 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:32.557029 10883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20 [sys.catalog]: This master's current role is: LEADER
W20240501 19:57:32.559227 10896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:57:32.559285 10896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:57:32.559341 10898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:32.560012 10898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:32.563791 10898 catalog_manager.cc:1347] Generated new cluster ID: 381f6bc046644938b8998405b8da8314
I20240501 19:57:32.563848 10898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:32.577443 10898 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:32.578389 10898 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:32.599241 10898 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 06cf8280f20843729e99a32633f62d20: Generated new TSK 0
I20240501 19:57:32.599737 10898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:32.603022 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.65:0
--local_ip_for_outbound_sockets=127.30.1.65
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:57:32.701776 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:32.701956 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:32.702025 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:32.704856 10901 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:32.704946 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:32.705053 10901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.65
I20240501 19:57:32.708451 10901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.1.65
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.10901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.65
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:32.709199 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:32.710129 10901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:32.717433 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.717902 10907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.718609 10909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:32.719319 10901 server_base.cc:1034] running on GCE node
I20240501 19:57:32.719628 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:32.720064 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:32.721199 10901 hybrid_clock.cc:648] HybridClock initialized: now 1714593452721175 us; error 35 us; skew 500 ppm
I20240501 19:57:32.722844 10901 webserver.cc:460] Webserver started at http://127.30.1.65:42735/ using document root <none> and password file <none>
I20240501 19:57:32.723372 10901 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:32.723434 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:32.723603 10901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:32.725056 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "99348cf9a56645808d00c459fb76a6f8"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.725481 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "99348cf9a56645808d00c459fb76a6f8"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.728307 10901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:32.730135 10915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.730983 10901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:32.731069 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "99348cf9a56645808d00c459fb76a6f8"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.731150 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:32.759752 10901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:32.760172 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:32.760310 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:32.760720 10901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:32.761444 10901 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:32.761489 10901 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.761521 10901 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:32.761540 10901 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.769034 10901 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.65:40027
I20240501 19:57:32.769054 11028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.65:40027 every 8 connection(s)
I20240501 19:57:32.769722 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:57:32.777336 11029 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:32.777511 11029 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:32.777879 11029 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:32.778224 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 10901
I20240501 19:57:32.778364 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:57:32.779160 10841 ts_manager.cc:194] Registered new tserver with Master: 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:32.780217 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.65:48759
I20240501 19:57:32.780272 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.66:0
--local_ip_for_outbound_sockets=127.30.1.66
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:57:32.875917 11032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:32.876081 11032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:32.876118 11032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:32.878824 11032 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:32.878875 11032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:32.878937 11032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.66
I20240501 19:57:32.882299 11032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.1.66
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.66
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:32.882908 11032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:32.883803 11032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:32.890995 11040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.891724 11038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:32.892381 11037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:32.893117 11032 server_base.cc:1034] running on GCE node
I20240501 19:57:32.893410 11032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:32.893858 11032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:32.895011 11032 hybrid_clock.cc:648] HybridClock initialized: now 1714593452895000 us; error 24 us; skew 500 ppm
I20240501 19:57:32.896472 11032 webserver.cc:460] Webserver started at http://127.30.1.66:46369/ using document root <none> and password file <none>
I20240501 19:57:32.896929 11032 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:32.897014 11032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:32.897179 11032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:32.898572 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.898986 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.901870 11032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:32.903715 11046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.904498 11032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:32.904590 11032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
format_stamp: "Formatted at 2024-05-01 19:57:32 on dist-test-slave-htmr"
I20240501 19:57:32.904664 11032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:32.934947 11032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:32.935364 11032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:32.935495 11032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:32.935917 11032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:32.936630 11032 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:32.936676 11032 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.936728 11032 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:32.936758 11032 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:32.944352 11032 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.66:38739
I20240501 19:57:32.944375 11159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.66:38739 every 8 connection(s)
I20240501 19:57:32.945122 11032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:57:32.945412 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11032
I20240501 19:57:32.945530 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:57:32.947476 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.67:0
--local_ip_for_outbound_sockets=127.30.1.67
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:32.954042 11160 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:32.954214 11160 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:32.954567 11160 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:32.955423 10841 ts_manager.cc:194] Registered new tserver with Master: 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:32.956171 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.66:52799
W20240501 19:57:33.045869 11163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.046028 11163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.046064 11163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.048863 11163 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.048926 11163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.048986 11163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.67
I20240501 19:57:33.052255 11163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.1.67
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.67
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.052889 11163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.053813 11163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.061357 11171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.062112 11169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.062613 11168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.063316 11163 server_base.cc:1034] running on GCE node
I20240501 19:57:33.063611 11163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.064059 11163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.065196 11163 hybrid_clock.cc:648] HybridClock initialized: now 1714593453065189 us; error 23 us; skew 500 ppm
I20240501 19:57:33.066843 11163 webserver.cc:460] Webserver started at http://127.30.1.67:36647/ using document root <none> and password file <none>
I20240501 19:57:33.067314 11163 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.067366 11163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.067513 11163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.068840 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.069314 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.072180 11163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:33.074157 11177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.075055 11163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:33.075181 11163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.075258 11163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.090395 11163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.090850 11163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.090988 11163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.091418 11163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.092180 11163 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.092227 11163 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.092279 11163 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.092310 11163 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.100050 11163 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.67:33555
I20240501 19:57:33.100075 11290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.67:33555 every 8 connection(s)
I20240501 19:57:33.100854 11163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:57:33.102911 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11163
I20240501 19:57:33.103031 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:57:33.105141 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.68:0
--local_ip_for_outbound_sockets=127.30.1.68
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:33.109138 11291 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.109330 11291 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.109704 11291 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.110519 10841 ts_manager.cc:194] Registered new tserver with Master: 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:33.111007 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.67:46155
W20240501 19:57:33.202327 11294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.202495 11294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.202541 11294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.205214 11294 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.205265 11294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.205360 11294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.68
I20240501 19:57:33.208657 11294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.1.68
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.68
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.209295 11294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.210198 11294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.217269 11299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.217743 11302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.218485 11300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.219164 11294 server_base.cc:1034] running on GCE node
I20240501 19:57:33.219455 11294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.219907 11294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.221055 11294 hybrid_clock.cc:648] HybridClock initialized: now 1714593453221037 us; error 30 us; skew 500 ppm
I20240501 19:57:33.227109 11294 webserver.cc:460] Webserver started at http://127.30.1.68:39091/ using document root <none> and password file <none>
I20240501 19:57:33.227516 11294 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.227560 11294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.227699 11294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.229099 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1679f84270414f67ac67c882220e1472"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.229516 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1679f84270414f67ac67c882220e1472"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.232604 11294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:33.234571 11308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.235424 11294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:33.235507 11294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1679f84270414f67ac67c882220e1472"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.235596 11294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.247627 11294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.248014 11294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.248153 11294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.248571 11294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.249311 11294 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.249354 11294 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.249394 11294 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.249428 11294 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.256759 11294 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.68:34371
I20240501 19:57:33.256814 11421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.68:34371 every 8 connection(s)
I20240501 19:57:33.257601 11294 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:57:33.260208 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11294
I20240501 19:57:33.260324 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:57:33.262408 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:0
--local_ip_for_outbound_sockets=127.30.1.69
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:33.265831 11422 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.266000 11422 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.266368 11422 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.267266 10841 ts_manager.cc:194] Registered new tserver with Master: 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:33.268040 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.68:46529
W20240501 19:57:33.362448 11425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.362619 11425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.362658 11425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.365485 11425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.365531 11425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.365603 11425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:57:33.368753 11425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.369414 11425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.370273 11425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.377601 11433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.378194 11430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.378922 11431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.379624 11425 server_base.cc:1034] running on GCE node
I20240501 19:57:33.379912 11425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.380327 11425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.381461 11425 hybrid_clock.cc:648] HybridClock initialized: now 1714593453381439 us; error 33 us; skew 500 ppm
I20240501 19:57:33.383064 11425 webserver.cc:460] Webserver started at http://127.30.1.69:33209/ using document root <none> and password file <none>
I20240501 19:57:33.383554 11425 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.383608 11425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.383744 11425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.385152 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "417972166f4d40ca9b4b6471a90be6af"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.385535 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "417972166f4d40ca9b4b6471a90be6af"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.388317 11425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:33.390190 11439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.391024 11425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:33.391109 11425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "417972166f4d40ca9b4b6471a90be6af"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.391222 11425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.404186 11425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.404596 11425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.404728 11425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.405164 11425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.405946 11425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.405992 11425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.406031 11425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.406064 11425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.413647 11425 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:38171
I20240501 19:57:33.413671 11552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:38171 every 8 connection(s)
I20240501 19:57:33.414477 11425 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:57:33.417872 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11425
I20240501 19:57:33.418007 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:57:33.420141 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:0
--local_ip_for_outbound_sockets=127.30.1.70
--webserver_interface=127.30.1.70
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:33.422797 11553 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.422996 11553 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.423375 11553 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.424160 10841 ts_manager.cc:194] Registered new tserver with Master: 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:33.424734 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:52413
W20240501 19:57:33.526680 11556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.526868 11556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.526934 11556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.529779 11556 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.529844 11556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.529927 11556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:57:33.533304 11556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.534068 11556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.535014 11556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.543377 11562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.544004 11561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.544775 11564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.545540 11556 server_base.cc:1034] running on GCE node
I20240501 19:57:33.545969 11556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.546517 11556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.547689 11556 hybrid_clock.cc:648] HybridClock initialized: now 1714593453547637 us; error 68 us; skew 500 ppm
I20240501 19:57:33.549799 11556 webserver.cc:460] Webserver started at http://127.30.1.70:41109/ using document root <none> and password file <none>
I20240501 19:57:33.550370 11556 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.550420 11556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.550582 11556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.552476 11556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "14002a0a4c674b438c78ca5db73637cc"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.553068 11556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "14002a0a4c674b438c78ca5db73637cc"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.557408 11556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:33.559377 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.560369 11556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:33.560464 11556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "14002a0a4c674b438c78ca5db73637cc"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.560544 11556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.572146 11556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.572602 11556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.572736 11556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.573230 11556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.574018 11556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.574066 11556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.574127 11556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.574153 11556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.583789 11556 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:35165
I20240501 19:57:33.583837 11683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:35165 every 8 connection(s)
I20240501 19:57:33.584592 11556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:57:33.585438 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11556
I20240501 19:57:33.585553 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:57:33.588274 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:0
--local_ip_for_outbound_sockets=127.30.1.71
--webserver_interface=127.30.1.71
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:33.596362 11684 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.596575 11684 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.597131 11684 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.598093 10841 ts_manager.cc:194] Registered new tserver with Master: 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 19:57:33.598800 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:40125
W20240501 19:57:33.705946 11687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.706120 11687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.706158 11687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.709070 11687 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.709123 11687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.709185 11687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:57:33.712509 11687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.713160 11687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.714010 11687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.721369 11695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.721822 11692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.722379 11693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.723127 11687 server_base.cc:1034] running on GCE node
I20240501 19:57:33.723440 11687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.723874 11687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.724987 11687 hybrid_clock.cc:648] HybridClock initialized: now 1714593453724963 us; error 34 us; skew 500 ppm
I20240501 19:57:33.726677 11687 webserver.cc:460] Webserver started at http://127.30.1.71:34171/ using document root <none> and password file <none>
I20240501 19:57:33.727118 11687 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.727167 11687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.727322 11687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.728617 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ea26a64181784350bcd3540d0633b219"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.729038 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ea26a64181784350bcd3540d0633b219"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.731868 11687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:33.733770 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.734704 11687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:33.734804 11687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ea26a64181784350bcd3540d0633b219"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.734879 11687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.759660 11687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.760095 11687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.760236 11687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.760632 11687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.761343 11687 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.761389 11687 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.761440 11687 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.761471 11687 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.769014 11687 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:32843
I20240501 19:57:33.769083 11814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:32843 every 8 connection(s)
I20240501 19:57:33.769876 11687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:57:33.776969 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11687
I20240501 19:57:33.777079 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:57:33.777381 11815 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.777557 11815 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.777971 11815 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.778795 10841 ts_manager.cc:194] Registered new tserver with Master: ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843)
I20240501 19:57:33.779045 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.72:0
--local_ip_for_outbound_sockets=127.30.1.72
--webserver_interface=127.30.1.72
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:33.779429 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:36989
I20240501 19:57:33.782402 11029 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
W20240501 19:57:33.876219 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:33.876375 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:33.876411 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:33.879246 11818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:33.879323 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:33.879429 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.72
I20240501 19:57:33.882802 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.1.72
--webserver_port=0
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.72
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:33.883481 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:33.884361 11818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:33.891479 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.892273 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:33.892761 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:33.893530 11818 server_base.cc:1034] running on GCE node
I20240501 19:57:33.893852 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:33.894302 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:33.895444 11818 hybrid_clock.cc:648] HybridClock initialized: now 1714593453895420 us; error 36 us; skew 500 ppm
I20240501 19:57:33.897053 11818 webserver.cc:460] Webserver started at http://127.30.1.72:39617/ using document root <none> and password file <none>
I20240501 19:57:33.897497 11818 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:33.897547 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:33.897707 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:33.899040 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.899453 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.902360 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:57:33.904250 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.905200 11818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:33.905290 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:33.905365 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:33.920771 11818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:33.921252 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:33.921382 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:33.921810 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:33.922534 11818 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:33.922583 11818 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.922633 11818 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:33.922665 11818 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:33.930418 11818 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.72:35171
I20240501 19:57:33.930441 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.72:35171 every 8 connection(s)
I20240501 19:57:33.931124 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:57:33.934110 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 11818
I20240501 19:57:33.934242 30725 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:57:33.939209 11946 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:33.939388 11946 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:33.939742 11946 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:33.940492 10841 ts_manager.cc:194] Registered new tserver with Master: e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:33.941020 10841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.72:35471
I20240501 19:57:33.947885 30725 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:57:33.956285 30725 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:57:33.956341 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 11425
I20240501 19:57:33.957767 11160 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:33.962628 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 11556
I20240501 19:57:33.968350 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 11687
I20240501 19:57:33.973311 30725 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu with pid 11818
I20240501 19:57:34.112691 11291 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:34.269718 11422 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:37.729885 10841 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:37.748569 11093 tablet_service.cc:1467] Processing CreateTablet for tablet 6edb640b8cc04d20b0815b1beb45d9c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.749076 11224 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8c5aab87341e3b79a8e121e6db01e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.749727 11093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.749691 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 19f79cd7d4ec4b4697a2637e6d14a892 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.749526 11356 tablet_service.cc:1467] Processing CreateTablet for tablet 6edb640b8cc04d20b0815b1beb45d9c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.749934 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8c5aab87341e3b79a8e121e6db01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.750094 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.750272 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.752960 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 19f79cd7d4ec4b4697a2637e6d14a892 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.753207 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.749529 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8c5aab87341e3b79a8e121e6db01e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.754633 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 19f79cd7d4ec4b4697a2637e6d14a892 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.755043 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8c5aab87341e3b79a8e121e6db01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.755371 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.757079 10962 tablet_service.cc:1467] Processing CreateTablet for tablet 6edb640b8cc04d20b0815b1beb45d9c8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.757158 11967 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:37.757313 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.757529 11966 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:37.757807 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc8c5aab87341e3b79a8e121e6db01e (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:37.758001 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8c5aab87341e3b79a8e121e6db01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.759088 11967 tablet_bootstrap.cc:654] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.759332 11966 tablet_bootstrap.cc:654] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.759877 11967 log.cc:824] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:37.760092 11966 log.cc:824] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:37.760602 11968 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:37.761523 11967 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:37.761673 11967 ts_tablet_manager.cc:1367] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:57:37.761708 11966 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:37.761854 11966 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:37.762388 11968 tablet_bootstrap.cc:654] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.763062 11968 log.cc:824] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:37.764461 11968 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:37.764614 11968 ts_tablet_manager.cc:1367] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:57:37.765076 11967 raft_consensus.cc:348] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.765257 11967 raft_consensus.cc:374] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.765326 11967 raft_consensus.cc:729] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.765336 11966 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.765507 11966 raft_consensus.cc:374] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.765551 11966 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.765983 11966 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.766508 11967 consensus_queue.cc:260] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.766881 11966 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:57:37.767221 11966 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:37.767477 11967 ts_tablet_manager.cc:1398] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:57:37.767803 11967 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:37.768077 11966 tablet_bootstrap.cc:654] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.767666 11971 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:37.768749 11967 tablet_bootstrap.cc:654] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.769815 11967 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:37.769946 11967 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:37.770444 11967 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.770503 11968 raft_consensus.cc:348] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.770619 11967 raft_consensus.cc:374] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.770666 11967 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.770685 11968 raft_consensus.cc:374] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.770820 11967 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.771190 11967 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:37.771350 11968 raft_consensus.cc:729] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.771775 11968 consensus_queue.cc:260] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.772481 11968 ts_tablet_manager.cc:1398] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 19:57:37.772784 11968 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:37.773655 11968 tablet_bootstrap.cc:654] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.774606 11968 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:37.774684 11968 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:37.775161 11968 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.775238 11968 raft_consensus.cc:374] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.775257 11968 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.775362 11968 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.775586 11968 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:37.777963 11971 tablet_bootstrap.cc:654] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.778590 11971 log.cc:824] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:37.779485 11966 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:37.779564 11966 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 19:57:37.779944 11966 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.780027 11966 raft_consensus.cc:374] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.780061 11966 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.780071 11971 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:37.780213 11971 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 19:57:37.780211 11966 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.780462 11966 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:37.782805 11971 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.782967 11971 raft_consensus.cc:374] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.783015 11971 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.783288 11971 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:37.783869 11971 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:57:37.784052 11971 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:37.784770 11971 tablet_bootstrap.cc:654] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.785687 11971 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:37.785766 11971 ts_tablet_manager.cc:1367] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:37.786162 11971 raft_consensus.cc:348] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.786247 11971 raft_consensus.cc:374] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.786279 11971 raft_consensus.cc:729] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.786393 11971 consensus_queue.cc:260] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.786677 11971 ts_tablet_manager.cc:1398] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:37.786882 11971 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:37.787529 11971 tablet_bootstrap.cc:654] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:37.788720 11971 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:37.788787 11971 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:37.789139 11971 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.789212 11971 raft_consensus.cc:374] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:37.789232 11971 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.789335 11971 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.789527 11971 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 19:57:37.854770 11292 tablet.cc:2369] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:37.909320 11978 raft_consensus.cc:482] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:37.909463 11978 raft_consensus.cc:504] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.910385 11978 leader_election.cc:288] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:37.913904 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8c5aab87341e3b79a8e121e6db01e" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:37.913906 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8c5aab87341e3b79a8e121e6db01e" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:37.914134 10983 raft_consensus.cc:2454] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:37.914129 11245 raft_consensus.cc:2454] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:37.914521 11312 leader_election.cc:302] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:37.914711 11978 raft_consensus.cc:2781] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:37.914790 11978 raft_consensus.cc:482] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:37.914839 11978 raft_consensus.cc:3037] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:37.915611 11978 raft_consensus.cc:504] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.915930 11978 leader_election.cc:288] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:37.916251 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8c5aab87341e3b79a8e121e6db01e" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:37.916297 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc8c5aab87341e3b79a8e121e6db01e" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:37.916369 11245 raft_consensus.cc:3037] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:37.916384 10983 raft_consensus.cc:3037] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:37.917392 10983 raft_consensus.cc:2454] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:37.917392 11245 raft_consensus.cc:2454] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:37.917660 11312 leader_election.cc:302] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:37.917888 11978 raft_consensus.cc:2781] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:37.918047 11978 raft_consensus.cc:686] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:37.918308 11978 consensus_queue.cc:237] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:37.920635 10841 catalog_manager.cc:5484] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
W20240501 19:57:37.949026 11161 tablet.cc:2369] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:57:38.012251 11423 tablet.cc:2369] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:57:38.023728 11030 tablet.cc:2369] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:38.025811 11973 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.025955 11973 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:38.026948 11973 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:38.030447 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:38.030484 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:38.030562 11245 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:38.030608 10983 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:38.030903 11048 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:38.031126 11973 raft_consensus.cc:2781] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.031193 11973 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.031239 11973 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.032008 11973 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:38.032310 11973 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:38.032642 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:38.032658 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:38.032753 10983 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.032775 11245 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.033675 10983 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:38.033703 11245 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:38.033931 11048 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:38.034104 11973 raft_consensus.cc:2781] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.034296 11973 raft_consensus.cc:686] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:38.034595 11973 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:38.037123 10841 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:38.047279 11978 raft_consensus.cc:482] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.047374 11978 raft_consensus.cc:504] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.047741 11978 leader_election.cc:288] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:38.048151 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edb640b8cc04d20b0815b1beb45d9c8" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:38.048282 10983 raft_consensus.cc:2454] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.048521 11312 leader_election.cc:302] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.048671 11978 raft_consensus.cc:2781] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.048734 11978 raft_consensus.cc:482] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.048794 11978 raft_consensus.cc:3037] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.049551 11978 raft_consensus.cc:504] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.049854 11978 leader_election.cc:288] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:38.050168 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edb640b8cc04d20b0815b1beb45d9c8" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:38.050261 10983 raft_consensus.cc:3037] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.051028 10983 raft_consensus.cc:2454] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.051321 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edb640b8cc04d20b0815b1beb45d9c8" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:38.051328 11312 leader_election.cc:302] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.051471 11114 raft_consensus.cc:2454] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.051532 11978 raft_consensus.cc:2781] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.051607 11978 raft_consensus.cc:686] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:38.051724 11978 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.051824 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edb640b8cc04d20b0815b1beb45d9c8" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:38.051916 11114 raft_consensus.cc:3037] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.052803 11114 raft_consensus.cc:2454] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.053076 10841 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 19:57:38.204530 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:38.204699 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:38.207448 11993 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:57:38.283975 10983 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.284592 11114 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.284847 10983 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.284808 11978 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.285202 11978 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.285423 11245 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.287437 11986 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.288237 11986 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.291661 12021 mvcc.cc:205] Tried to move back new op lower bound from 7022974805122912256 to 7022974804181106688. Current Snapshot: MvccSnapshot[applied={T|T < 7022974805122912256}]
I20240501 19:57:38.292316 11244 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.293227 11973 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.293870 10983 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.294240 11973 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.311774 12025 mvcc.cc:205] Tried to move back new op lower bound from 7022974805137997824 to 7022974804181106688. Current Snapshot: MvccSnapshot[applied={T|T < 7022974805122912256}]
I20240501 19:57:38.312718 12040 mvcc.cc:205] Tried to move back new op lower bound from 7022974805122912256 to 7022974804181106688. Current Snapshot: MvccSnapshot[applied={T|T < 7022974805122912256}]
I20240501 19:57:38.316105 12033 mvcc.cc:205] Tried to move back new op lower bound from 7022974805130268672 to 7022974803635445760. Current Snapshot: MvccSnapshot[applied={T|T < 7022974805130268672}]
INSERT report
rows total: 2000
time total: 81.2036 ms
time per row: 0.0406018 ms
I20240501 19:57:38.331122 10838 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:38.337122 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 19d8d7d255494a49920849d10b800fa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.337381 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.337914 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd6776aebf47c99f05850de83b36e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.338167 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.338232 11093 tablet_service.cc:1467] Processing CreateTablet for tablet 19d8d7d255494a49920849d10b800fa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.338461 11093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.340109 11356 tablet_service.cc:1467] Processing CreateTablet for tablet 19d8d7d255494a49920849d10b800fa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.340294 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd6776aebf47c99f05850de83b36e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.340349 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.340514 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.340703 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8c1a34cc94d29986289939b2159bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.340885 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.341147 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8c1a34cc94d29986289939b2159bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.341329 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.344193 11224 tablet_service.cc:1467] Processing CreateTablet for tablet 9fcd6776aebf47c99f05850de83b36e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.344439 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.345873 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8c1a34cc94d29986289939b2159bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.346104 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.349841 12054 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:38.350381 12057 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:38.350450 12056 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:38.351280 12056 tablet_bootstrap.cc:654] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.351482 12057 tablet_bootstrap.cc:654] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.352197 12055 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:38.352478 12056 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:38.352566 12056 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.352896 12057 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:38.353035 12057 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:38.353123 12055 tablet_bootstrap.cc:654] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.353273 12056 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.353389 12056 raft_consensus.cc:374] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.353421 12056 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.353494 12057 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.353600 12057 raft_consensus.cc:374] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.353560 12056 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.353636 12057 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.353812 12056 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:38.353751 12057 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.353852 12054 tablet_bootstrap.cc:654] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.353960 12056 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:38.354025 12057 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.354171 12057 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:38.354671 12056 tablet_bootstrap.cc:654] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.354931 12057 tablet_bootstrap.cc:654] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.355608 12056 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:38.355682 12056 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:38.356073 12056 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.356173 12056 raft_consensus.cc:374] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.356209 12056 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.356328 12056 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.356573 12056 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:38.356750 12056 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:38.357098 12057 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:38.357180 12057 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:38.357568 12056 tablet_bootstrap.cc:654] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.357539 12057 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.357637 12057 raft_consensus.cc:374] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.357686 12057 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.357800 12057 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.358040 12057 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.358914 12056 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:38.358989 12056 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.359252 12054 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:38.359328 12054 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:57:38.359345 12056 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.359452 12056 raft_consensus.cc:374] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.359489 12056 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.359637 12056 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.359817 12054 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.359907 12056 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.359928 12054 raft_consensus.cc:374] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.359962 12054 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.360105 12054 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.360396 12055 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:38.360472 12055 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:38.360564 12054 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.360710 12054 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:38.360885 12055 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.361009 12055 raft_consensus.cc:374] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.361042 12055 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.361146 12055 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.361443 12055 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:38.362087 12054 tablet_bootstrap.cc:654] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.363382 12054 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:38.363457 12054 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:38.363780 12054 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.363853 12054 raft_consensus.cc:374] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.363888 12054 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.364001 12054 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.364424 12054 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.364578 12054 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:38.365495 12054 tablet_bootstrap.cc:654] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.366374 12054 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:38.366464 12054 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.366816 12054 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.366890 12054 raft_consensus.cc:374] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.366909 12054 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.367012 12054 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.367224 12054 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:38.406028 11986 raft_consensus.cc:482] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.406108 11986 raft_consensus.cc:504] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.406481 11986 leader_election.cc:288] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.406795 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8c1a34cc94d29986289939b2159bb" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:38.406803 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8c1a34cc94d29986289939b2159bb" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:38.406891 11114 raft_consensus.cc:2454] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.406899 11244 raft_consensus.cc:2454] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.407172 11311 leader_election.cc:302] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:38.407362 11986 raft_consensus.cc:2781] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.407423 11986 raft_consensus.cc:482] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.407474 11986 raft_consensus.cc:3037] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.408365 11986 raft_consensus.cc:504] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.408609 11986 leader_election.cc:288] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.408876 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8c1a34cc94d29986289939b2159bb" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:38.408954 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8c1a34cc94d29986289939b2159bb" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:38.409040 11244 raft_consensus.cc:3037] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.409044 11114 raft_consensus.cc:3037] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.409755 11114 raft_consensus.cc:2454] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.410020 11311 leader_election.cc:302] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:38.410084 11244 raft_consensus.cc:2454] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.410183 11986 raft_consensus.cc:2781] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.410264 11986 raft_consensus.cc:686] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:38.410382 11986 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.411703 10837 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:38.518049 11981 raft_consensus.cc:482] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.518160 11981 raft_consensus.cc:504] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.519063 11981 leader_election.cc:288] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.522560 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcd6776aebf47c99f05850de83b36e6" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:38.522579 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcd6776aebf47c99f05850de83b36e6" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:38.522673 11244 raft_consensus.cc:2454] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:38.522683 11114 raft_consensus.cc:2454] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:38.523319 10918 leader_election.cc:302] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.523542 11981 raft_consensus.cc:2781] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.523594 11981 raft_consensus.cc:482] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.523634 11981 raft_consensus.cc:3037] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.524356 11981 raft_consensus.cc:504] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.524680 11981 leader_election.cc:288] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.524997 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcd6776aebf47c99f05850de83b36e6" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:38.525110 11114 raft_consensus.cc:3037] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.525818 11114 raft_consensus.cc:2454] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:38.526006 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fcd6776aebf47c99f05850de83b36e6" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:38.526088 11244 raft_consensus.cc:3037] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.526067 10918 leader_election.cc:302] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.526229 11981 raft_consensus.cc:2781] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.526386 11981 raft_consensus.cc:686] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:38.526633 11981 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.526872 11244 raft_consensus.cc:2454] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:38.528204 10837 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:38.536863 11978 raft_consensus.cc:482] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.536976 11978 raft_consensus.cc:504] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.537232 11978 leader_election.cc:288] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.537508 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:38.537509 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:38.537590 11244 raft_consensus.cc:2454] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.537600 11114 raft_consensus.cc:2454] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:38.537815 11310 leader_election.cc:302] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 19:57:38.537984 11978 raft_consensus.cc:2781] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.538033 11978 raft_consensus.cc:482] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.538057 11978 raft_consensus.cc:3037] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.538657 11978 raft_consensus.cc:504] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.538928 11978 leader_election.cc:288] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:38.539238 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:38.539320 11114 raft_consensus.cc:3037] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.539392 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:38.539484 11244 raft_consensus.cc:3037] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.540053 11114 raft_consensus.cc:2454] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.540115 11244 raft_consensus.cc:2454] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:38.540307 11311 leader_election.cc:302] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:38.540539 11978 raft_consensus.cc:2781] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.540633 11978 raft_consensus.cc:686] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:38.540776 11978 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.545797 10837 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
W20240501 19:57:38.673914 12073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:38.674103 12073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:38.676954 12073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:57:38.742650 11114 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.742717 11244 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.743647 11114 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.743606 11978 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.743714 11244 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.744169 11978 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.744670 11978 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.745088 11978 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.763219 11244 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.763659 12070 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:38.764869 11114 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:38.765530 12070 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.7366 ms
time per row: 0.0478683 ms
I20240501 19:57:38.810544 10836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:38.815275 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.815382 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.815546 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.815584 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.816097 11356 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.816205 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.816365 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.816397 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.817799 12056 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:38.818272 12057 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:38.818432 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.818509 12056 tablet_bootstrap.cc:654] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.818619 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e48d711c5714ee58ebbcbd094ebf3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.819043 12055 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:38.819144 12057 tablet_bootstrap.cc:654] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.819880 12055 tablet_bootstrap.cc:654] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.820209 12057 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:38.820183 11354 tablet_service.cc:1467] Processing CreateTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.820277 12057 ts_tablet_manager.cc:1367] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.820354 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.820676 12057 raft_consensus.cc:348] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.820789 12057 raft_consensus.cc:374] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.820829 12057 raft_consensus.cc:729] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.820951 12057 consensus_queue.cc:260] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.821228 12057 ts_tablet_manager.cc:1398] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.821190 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.821384 12057 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:38.821449 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e48d711c5714ee58ebbcbd094ebf3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.822197 12057 tablet_bootstrap.cc:654] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.823249 12054 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:38.823556 12056 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:38.823630 12056 ts_tablet_manager.cc:1367] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:57:38.823940 12056 raft_consensus.cc:348] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.824007 12054 tablet_bootstrap.cc:654] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.824021 12056 raft_consensus.cc:374] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.824054 12056 raft_consensus.cc:729] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.824179 12056 consensus_queue.cc:260] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.824433 12056 ts_tablet_manager.cc:1398] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:38.824797 12056 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:38.824829 12055 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:38.824796 10962 tablet_service.cc:1467] Processing CreateTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.824923 12055 ts_tablet_manager.cc:1367] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:57:38.825026 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.825342 12055 raft_consensus.cc:348] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.825456 12055 raft_consensus.cc:374] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.825488 12055 raft_consensus.cc:729] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.825589 12056 tablet_bootstrap.cc:654] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.825598 12055 consensus_queue.cc:260] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.825918 12055 ts_tablet_manager.cc:1398] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:38.826820 12056 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:38.826895 12056 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.826865 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:38.827081 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e48d711c5714ee58ebbcbd094ebf3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:38.827328 12056 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.827416 12056 raft_consensus.cc:374] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.827447 12056 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.827560 12056 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.827787 12056 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:38.828094 12055 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:38.828398 12054 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:38.828470 12054 ts_tablet_manager.cc:1367] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:38.828819 12054 raft_consensus.cc:348] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.828943 12054 raft_consensus.cc:374] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.828974 12054 raft_consensus.cc:729] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.829079 12054 consensus_queue.cc:260] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.829330 12054 ts_tablet_manager.cc:1398] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.829473 12055 tablet_bootstrap.cc:654] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.830158 12057 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:38.830235 12057 ts_tablet_manager.cc:1367] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:57:38.830463 12055 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:38.830564 12055 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:38.830670 12057 raft_consensus.cc:348] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.830768 12057 raft_consensus.cc:374] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.830803 12057 raft_consensus.cc:729] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.830909 12057 consensus_queue.cc:260] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.831149 12055 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.831223 12057 ts_tablet_manager.cc:1398] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.831247 12055 raft_consensus.cc:374] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.831279 12055 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.831372 12057 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:38.831414 12055 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.832233 12055 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:38.832408 12055 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:38.832949 12057 tablet_bootstrap.cc:654] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.833505 12055 tablet_bootstrap.cc:654] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:38.834033 12057 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:38.834143 12057 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:38.834616 12057 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.834720 12057 raft_consensus.cc:374] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.834792 12057 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.834939 12057 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:38.835212 12057 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:38.835517 12055 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:38.835597 12055 ts_tablet_manager.cc:1367] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:38.835995 12055 raft_consensus.cc:348] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.836089 12055 raft_consensus.cc:374] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:38.836143 12055 raft_consensus.cc:729] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:38.836257 12055 consensus_queue.cc:260] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:38.836573 12055 ts_tablet_manager.cc:1398] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:38.930176 11981 raft_consensus.cc:482] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.930264 11981 raft_consensus.cc:504] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.930655 11981 leader_election.cc:288] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:38.930972 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1bbe74dbc74fa687b0a64d3aa203f3" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:38.931109 11114 raft_consensus.cc:2454] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:38.931401 10918 leader_election.cc:302] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.931552 11981 raft_consensus.cc:2781] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:38.931607 11981 raft_consensus.cc:482] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.931638 11981 raft_consensus.cc:3037] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.932356 11981 raft_consensus.cc:504] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.932665 11981 leader_election.cc:288] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:38.932962 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1bbe74dbc74fa687b0a64d3aa203f3" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:38.933063 11114 raft_consensus.cc:3037] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:38.933746 11114 raft_consensus.cc:2454] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:38.934018 10918 leader_election.cc:302] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:38.934199 11981 raft_consensus.cc:2781] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:38.934298 11981 raft_consensus.cc:686] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:38.934391 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1bbe74dbc74fa687b0a64d3aa203f3" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:38.934470 11981 consensus_queue.cc:237] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:38.934389 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1bbe74dbc74fa687b0a64d3aa203f3" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:38.934605 11376 raft_consensus.cc:2454] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:38.935777 10836 catalog_manager.cc:5484] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:39.015393 11981 raft_consensus.cc:482] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.015504 11981 raft_consensus.cc:504] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.015869 11981 leader_election.cc:288] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.016242 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e48d711c5714ee58ebbcbd094ebf3e7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:39.016259 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e48d711c5714ee58ebbcbd094ebf3e7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.016366 11245 raft_consensus.cc:2454] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:39.016371 11375 raft_consensus.cc:2454] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:39.016770 10919 leader_election.cc:302] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.016973 11981 raft_consensus.cc:2781] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.017037 11981 raft_consensus.cc:482] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.017083 11981 raft_consensus.cc:3037] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.017819 11981 raft_consensus.cc:504] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.018122 11981 leader_election.cc:288] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.018399 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e48d711c5714ee58ebbcbd094ebf3e7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:39.018414 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e48d711c5714ee58ebbcbd094ebf3e7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.018505 11375 raft_consensus.cc:3037] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.018507 11245 raft_consensus.cc:3037] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.019208 11245 raft_consensus.cc:2454] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:39.019250 11375 raft_consensus.cc:2454] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:39.019470 10917 leader_election.cc:302] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.019624 11981 raft_consensus.cc:2781] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.019706 11981 raft_consensus.cc:686] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:39.019842 11981 consensus_queue.cc:237] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.021221 10836 catalog_manager.cc:5484] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:39.094033 11986 raft_consensus.cc:482] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.094142 11986 raft_consensus.cc:504] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.094528 11986 leader_election.cc:288] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:39.094828 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:39.094892 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:39.094944 10982 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.095016 11114 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.095363 11312 leader_election.cc:302] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.095539 11986 raft_consensus.cc:2781] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.095602 11986 raft_consensus.cc:482] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.095649 11986 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.096351 11986 raft_consensus.cc:504] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.096627 11986 leader_election.cc:288] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:39.096961 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:39.096985 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:39.097062 10982 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.097074 11114 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.097844 11114 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.097844 10982 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.098121 11311 leader_election.cc:302] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:39.098264 11986 raft_consensus.cc:2781] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.098352 11986 raft_consensus.cc:686] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:39.098505 11986 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.099884 10836 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 19:57:39.208972 12119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:39.209141 12119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:39.211952 12119 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:57:39.288043 11114 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.288241 10982 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.288427 11986 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.288661 11978 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.293892 11375 raft_consensus.cc:3037] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.294739 11375 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.295347 11245 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.295361 11981 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.296020 12070 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.296962 11375 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.298779 11981 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.300475 11113 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.301620 12070 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 89.2763 ms
time per row: 0.0446382 ms
I20240501 19:57:39.341378 10836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:39.346674 11225 tablet_service.cc:1467] Processing CreateTablet for tablet c49b9c5ebaab42e18fe7bbe8127d569e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.346930 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49b9c5ebaab42e18fe7bbe8127d569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.346921 11092 tablet_service.cc:1467] Processing CreateTablet for tablet c49b9c5ebaab42e18fe7bbe8127d569e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.347157 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49b9c5ebaab42e18fe7bbe8127d569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.347594 11354 tablet_service.cc:1467] Processing CreateTablet for tablet a170766f497f491ba9b45f92f2a0e751 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.347679 10963 tablet_service.cc:1467] Processing CreateTablet for tablet c49b9c5ebaab42e18fe7bbe8127d569e (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.347801 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a170766f497f491ba9b45f92f2a0e751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.347891 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49b9c5ebaab42e18fe7bbe8127d569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.349339 11094 tablet_service.cc:1467] Processing CreateTablet for tablet b4f0699b5bdf44499c87ff1541cd2ad3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.349586 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f0699b5bdf44499c87ff1541cd2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.349777 11225 tablet_service.cc:1467] Processing CreateTablet for tablet a170766f497f491ba9b45f92f2a0e751 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.349982 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a170766f497f491ba9b45f92f2a0e751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.349939 11355 tablet_service.cc:1467] Processing CreateTablet for tablet b4f0699b5bdf44499c87ff1541cd2ad3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.350142 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f0699b5bdf44499c87ff1541cd2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.351501 11224 tablet_service.cc:1467] Processing CreateTablet for tablet b4f0699b5bdf44499c87ff1541cd2ad3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.351699 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f0699b5bdf44499c87ff1541cd2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.352061 12174 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:39.352145 10962 tablet_service.cc:1467] Processing CreateTablet for tablet a170766f497f491ba9b45f92f2a0e751 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.352365 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a170766f497f491ba9b45f92f2a0e751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.353106 12174 tablet_bootstrap.cc:654] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.354102 12174 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:39.354197 12174 ts_tablet_manager.cc:1367] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:39.354650 12174 raft_consensus.cc:348] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.354749 12174 raft_consensus.cc:374] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.354790 12174 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.354908 12174 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.355018 12175 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:39.355163 12174 ts_tablet_manager.cc:1398] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.355304 12174 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:39.355859 12176 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:39.356050 12174 tablet_bootstrap.cc:654] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.356783 12176 tablet_bootstrap.cc:654] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.356956 12175 tablet_bootstrap.cc:654] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.357506 12177 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:39.357677 12174 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:39.357753 12174 ts_tablet_manager.cc:1367] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:39.358067 12174 raft_consensus.cc:348] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.358142 12174 raft_consensus.cc:374] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.358161 12174 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.358196 12177 tablet_bootstrap.cc:654] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.358242 12174 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.358451 12174 ts_tablet_manager.cc:1398] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.358592 12174 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:39.359434 12174 tablet_bootstrap.cc:654] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.359705 12177 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:39.359783 12177 ts_tablet_manager.cc:1367] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:39.360368 12174 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:39.360493 12174 ts_tablet_manager.cc:1367] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:39.360448 12177 raft_consensus.cc:348] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.360601 12177 raft_consensus.cc:374] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.360669 12177 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.360864 12177 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.361142 12174 raft_consensus.cc:348] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.361266 12174 raft_consensus.cc:374] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.361335 12177 ts_tablet_manager.cc:1398] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.361348 12176 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:39.361333 12174 raft_consensus.cc:729] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.361413 12176 ts_tablet_manager.cc:1367] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 19:57:39.361573 12174 consensus_queue.cc:260] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.361842 12176 raft_consensus.cc:348] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.361959 12176 raft_consensus.cc:374] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.361996 12176 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.362107 12174 ts_tablet_manager.cc:1398] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:39.362140 12177 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:39.362152 12176 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.362519 12176 ts_tablet_manager.cc:1398] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:39.362675 12176 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:39.363492 12176 tablet_bootstrap.cc:654] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.364523 12177 tablet_bootstrap.cc:654] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.364533 12176 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:39.364606 12176 ts_tablet_manager.cc:1367] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:39.365082 12176 raft_consensus.cc:348] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.365203 12176 raft_consensus.cc:374] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.365247 12176 raft_consensus.cc:729] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.365348 12176 consensus_queue.cc:260] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.365442 12175 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:39.365482 12177 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:39.365509 12175 ts_tablet_manager.cc:1367] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:39.365561 12177 ts_tablet_manager.cc:1367] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:57:39.365577 12176 ts_tablet_manager.cc:1398] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.365821 12175 raft_consensus.cc:348] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.365897 12175 raft_consensus.cc:374] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.365927 12175 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.365895 12177 raft_consensus.cc:348] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.365978 12177 raft_consensus.cc:374] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.366022 12177 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.366140 12177 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.366382 12177 ts_tablet_manager.cc:1398] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:39.366878 12175 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.367467 12175 ts_tablet_manager.cc:1398] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:39.367623 12175 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:39.368250 12175 tablet_bootstrap.cc:654] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.369419 12175 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:39.369495 12175 ts_tablet_manager.cc:1367] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:39.369791 12175 raft_consensus.cc:348] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.369884 12175 raft_consensus.cc:374] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.369936 12175 raft_consensus.cc:729] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.370045 12175 consensus_queue.cc:260] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.370296 12175 ts_tablet_manager.cc:1398] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.412335 11978 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.412412 11978 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.412735 11978 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:39.413074 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.413071 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:39.413169 10982 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.413178 11243 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.413437 11312 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.413587 11978 raft_consensus.cc:2781] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.413655 11978 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.413697 11978 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.414311 11978 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.414543 11978 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:39.414774 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.414815 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:39.414875 11243 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.414886 10982 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.415517 11243 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.415529 10982 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.415793 11310 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 19:57:39.415925 11978 raft_consensus.cc:2781] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.416009 11978 raft_consensus.cc:686] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:39.416129 11978 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:39.417363 10839 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:39.449541 12165 raft_consensus.cc:482] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.449641 12165 raft_consensus.cc:504] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.450058 12165 leader_election.cc:288] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:39.450387 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f0699b5bdf44499c87ff1541cd2ad3" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.450526 11243 raft_consensus.cc:2454] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:39.450800 11048 leader_election.cc:302] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:39.450971 12165 raft_consensus.cc:2781] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.451045 12165 raft_consensus.cc:482] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.451093 12165 raft_consensus.cc:3037] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.451913 12165 raft_consensus.cc:504] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.452214 12165 leader_election.cc:288] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:39.452495 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f0699b5bdf44499c87ff1541cd2ad3" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.452576 11243 raft_consensus.cc:3037] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.453223 11243 raft_consensus.cc:2454] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:39.453538 11048 leader_election.cc:302] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:39.453697 12165 raft_consensus.cc:2781] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.453773 12165 raft_consensus.cc:686] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:39.453900 12165 consensus_queue.cc:237] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.454054 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f0699b5bdf44499c87ff1541cd2ad3" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:39.454123 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f0699b5bdf44499c87ff1541cd2ad3" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:39.454187 11376 raft_consensus.cc:2454] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:39.455292 10839 catalog_manager.cc:5484] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:39.548063 12165 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.548177 12165 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.548501 12165 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:39.548833 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.548825 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:39.548970 11243 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:39.548967 10982 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:39.549201 11048 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:39.549351 12165 raft_consensus.cc:2781] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.549412 12165 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.549470 12165 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.550189 12165 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.550446 12165 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:39.550711 11243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.550762 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:39.550803 11243 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.550853 10982 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.551465 11243 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:39.551477 10982 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:39.551712 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.551882 12165 raft_consensus.cc:2781] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.551960 12165 raft_consensus.cc:686] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:39.552102 12165 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.553526 10839 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 19:57:39.681068 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:39.681250 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:39.684429 12191 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:57:39.751881 11243 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.752151 11375 raft_consensus.cc:3037] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.752267 11973 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.752802 10982 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.753103 11245 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.753224 11375 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.753463 12165 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.753609 11973 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.756361 11973 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.756976 10982 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.757330 11978 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:39.761780 11244 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:39.763046 11986 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 89.9704 ms
time per row: 0.0449852 ms
I20240501 19:57:39.814013 10836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:39.818769 10962 tablet_service.cc:1467] Processing CreateTablet for tablet d5faf889870246b78b82e19c041dc119 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.818874 11092 tablet_service.cc:1467] Processing CreateTablet for tablet d5faf889870246b78b82e19c041dc119 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.819010 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5faf889870246b78b82e19c041dc119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.819103 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5faf889870246b78b82e19c041dc119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.820602 12175 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:39.821003 12177 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:39.821460 12175 tablet_bootstrap.cc:654] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.821442 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.821671 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.821749 12177 tablet_bootstrap.cc:654] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.822832 12177 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:39.822912 12177 ts_tablet_manager.cc:1367] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:39.823298 12177 raft_consensus.cc:348] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.823381 12177 raft_consensus.cc:374] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.823406 12177 raft_consensus.cc:729] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.823510 12177 consensus_queue.cc:260] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.823817 12177 ts_tablet_manager.cc:1398] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.823951 12177 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:39.824028 11354 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.824273 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.824661 12177 tablet_bootstrap.cc:654] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.824981 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 26b12620ad9a404d9ab70e8506ad1a39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.825176 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.826087 11224 tablet_service.cc:1467] Processing CreateTablet for tablet d5faf889870246b78b82e19c041dc119 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.826262 12176 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:39.826303 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5faf889870246b78b82e19c041dc119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.827096 12176 tablet_bootstrap.cc:654] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.827380 10962 tablet_service.cc:1467] Processing CreateTablet for tablet 26b12620ad9a404d9ab70e8506ad1a39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.827591 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.824028 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 26b12620ad9a404d9ab70e8506ad1a39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.827767 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.828312 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea (DEFAULT_TABLE table=rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:39.828516 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:39.828999 12174 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:39.829818 12174 tablet_bootstrap.cc:654] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.829921 12176 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:39.829998 12176 ts_tablet_manager.cc:1367] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:39.830402 12176 raft_consensus.cc:348] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.830487 12176 raft_consensus.cc:374] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.830520 12176 raft_consensus.cc:729] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.830646 12176 consensus_queue.cc:260] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.830932 12176 ts_tablet_manager.cc:1398] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.831070 12176 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:39.831229 12174 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:39.831302 12174 ts_tablet_manager.cc:1367] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:39.831751 12174 raft_consensus.cc:348] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.831851 12174 raft_consensus.cc:374] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.831885 12174 raft_consensus.cc:729] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.831897 12176 tablet_bootstrap.cc:654] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.831986 12174 consensus_queue.cc:260] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.832285 12174 ts_tablet_manager.cc:1398] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.832445 12174 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:39.833200 12176 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:39.833290 12176 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:39.833389 12175 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:39.833469 12175 ts_tablet_manager.cc:1367] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240501 19:57:39.833506 12174 tablet_bootstrap.cc:654] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.833669 12176 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.833742 12176 raft_consensus.cc:374] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.833770 12176 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.833832 12175 raft_consensus.cc:348] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.833910 12175 raft_consensus.cc:374] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.833868 12176 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.833937 12175 raft_consensus.cc:729] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.834033 12175 consensus_queue.cc:260] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.834112 12176 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.834297 12175 ts_tablet_manager.cc:1398] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.834765 12175 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:39.835573 12175 tablet_bootstrap.cc:654] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.835649 12174 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:39.835733 12174 ts_tablet_manager.cc:1367] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:39.835855 12177 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:39.835932 12177 ts_tablet_manager.cc:1367] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240501 19:57:39.836185 12174 raft_consensus.cc:348] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.836316 12174 raft_consensus.cc:374] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.836318 12177 raft_consensus.cc:348] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.836377 12174 raft_consensus.cc:729] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.836416 12177 raft_consensus.cc:374] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.836449 12177 raft_consensus.cc:729] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.836508 12174 consensus_queue.cc:260] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.836546 12177 consensus_queue.cc:260] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.836787 12177 ts_tablet_manager.cc:1398] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:39.837143 12174 ts_tablet_manager.cc:1398] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.838163 12175 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:39.838244 12175 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:39.838407 12177 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:39.838868 12175 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.838958 12175 raft_consensus.cc:374] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.838987 12175 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.839136 12177 tablet_bootstrap.cc:654] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:39.839095 12175 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.839342 12175 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.840282 12177 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:39.840370 12177 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:39.840780 12177 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.840941 12177 raft_consensus.cc:374] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:39.840991 12177 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:39.841120 12177 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:39.841480 12177 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.844360 12232 raft_consensus.cc:482] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.844470 12232 raft_consensus.cc:504] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.844776 12232 leader_election.cc:288] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.845036 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5faf889870246b78b82e19c041dc119" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:39.845131 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5faf889870246b78b82e19c041dc119" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.845156 11114 raft_consensus.cc:2454] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:39.845234 11242 raft_consensus.cc:2454] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:39.845405 10918 leader_election.cc:302] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.845609 12232 raft_consensus.cc:2781] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.845674 12232 raft_consensus.cc:482] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.845724 12232 raft_consensus.cc:3037] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.846478 12232 raft_consensus.cc:504] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.846729 12232 leader_election.cc:288] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.847009 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5faf889870246b78b82e19c041dc119" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.846997 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5faf889870246b78b82e19c041dc119" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:39.847110 11114 raft_consensus.cc:3037] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.847121 11242 raft_consensus.cc:3037] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.847882 11242 raft_consensus.cc:2454] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:39.847882 11114 raft_consensus.cc:2454] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:39.848134 10917 leader_election.cc:302] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.848282 12232 raft_consensus.cc:2781] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.848363 12232 raft_consensus.cc:686] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:39.848507 12232 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:39.849815 10835 catalog_manager.cc:5484] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:39.966717 11978 raft_consensus.cc:482] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:39.966811 11978 raft_consensus.cc:504] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.967145 11978 leader_election.cc:288] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.967391 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb1f0dcd4cf49cd969eb28f1eb5d9ea" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:39.967479 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb1f0dcd4cf49cd969eb28f1eb5d9ea" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:39.967528 10983 raft_consensus.cc:2454] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.967586 11242 raft_consensus.cc:2454] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:39.967765 11312 leader_election.cc:302] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.967907 11978 raft_consensus.cc:2781] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:39.967964 11978 raft_consensus.cc:482] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:39.968004 11978 raft_consensus.cc:3037] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.968701 11978 raft_consensus.cc:504] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.969007 11978 leader_election.cc:288] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:39.969266 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb1f0dcd4cf49cd969eb28f1eb5d9ea" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:39.969374 10983 raft_consensus.cc:3037] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.969372 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb1f0dcd4cf49cd969eb28f1eb5d9ea" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:39.969489 11242 raft_consensus.cc:3037] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:39.970046 10983 raft_consensus.cc:2454] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.970197 11242 raft_consensus.cc:2454] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:39.970293 11312 leader_election.cc:302] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:39.970458 11978 raft_consensus.cc:2781] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:39.970525 11978 raft_consensus.cc:686] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:39.970670 11978 consensus_queue.cc:237] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:39.972093 10835 catalog_manager.cc:5484] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:40.023080 11973 raft_consensus.cc:482] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.023178 11973 raft_consensus.cc:504] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.023504 11973 leader_election.cc:288] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.023876 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b12620ad9a404d9ab70e8506ad1a39" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:40.023903 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b12620ad9a404d9ab70e8506ad1a39" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:40.023994 10983 raft_consensus.cc:2454] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.024040 11375 raft_consensus.cc:2454] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.024274 11050 leader_election.cc:302] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.024432 11973 raft_consensus.cc:2781] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.024487 11973 raft_consensus.cc:482] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.024538 11973 raft_consensus.cc:3037] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.025337 11973 raft_consensus.cc:504] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.025604 11973 leader_election.cc:288] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.025885 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b12620ad9a404d9ab70e8506ad1a39" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:40.025972 10983 raft_consensus.cc:3037] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.026032 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b12620ad9a404d9ab70e8506ad1a39" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:40.026136 11375 raft_consensus.cc:3037] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.026665 10983 raft_consensus.cc:2454] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.026846 11375 raft_consensus.cc:2454] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.026922 11050 leader_election.cc:302] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.027082 11973 raft_consensus.cc:2781] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.027184 11973 raft_consensus.cc:686] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:40.027294 11973 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.028781 10835 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 19:57:40.158138 12243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:40.158334 12243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:40.161191 12243 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:57:40.251813 11114 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.252249 12232 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.253150 11242 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.256824 11242 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.257666 11375 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.257823 11978 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.257966 10983 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.258416 11973 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.258456 10983 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.258651 11973 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.258803 11978 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.259984 12231 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.291280 12146 mvcc.cc:205] Tried to move back new op lower bound from 7022974813322190848 to 7022974812040822784. Current Snapshot: MvccSnapshot[applied={T|T < 7022974813207289856}]
I20240501 19:57:40.298791 12152 mvcc.cc:205] Tried to move back new op lower bound from 7022974813186113536 to 7022974811540398080. Current Snapshot: MvccSnapshot[applied={T|T < 7022974813186113536}]
I20240501 19:57:40.303562 12156 mvcc.cc:205] Tried to move back new op lower bound from 7022974813186113536 to 7022974811540398080. Current Snapshot: MvccSnapshot[applied={T|T < 7022974813186113536}]
INSERT report
rows total: 2000
time total: 126.139 ms
time per row: 0.0630697 ms
I20240501 19:57:40.327119 10833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:40.331919 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 219742bd987e4a1990233fa8c972c232 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.332018 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 219742bd987e4a1990233fa8c972c232 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.332209 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219742bd987e4a1990233fa8c972c232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.332191 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 219742bd987e4a1990233fa8c972c232 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.332288 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219742bd987e4a1990233fa8c972c232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.332432 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219742bd987e4a1990233fa8c972c232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.333128 11224 tablet_service.cc:1467] Processing CreateTablet for tablet 5d032fb77ac944629a45b808cc76c2f1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.333372 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d032fb77ac944629a45b808cc76c2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.334539 12176 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:40.334637 12175 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:40.335224 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd522180bd4e53906cd9572f82ab67 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.335371 12176 tablet_bootstrap.cc:654] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.335371 12175 tablet_bootstrap.cc:654] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.335458 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd522180bd4e53906cd9572f82ab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.336514 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 5d032fb77ac944629a45b808cc76c2f1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.336719 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d032fb77ac944629a45b808cc76c2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.336714 10962 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd522180bd4e53906cd9572f82ab67 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.336891 12175 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:40.336989 12175 ts_tablet_manager.cc:1367] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:40.337010 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd522180bd4e53906cd9572f82ab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.337332 12174 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:40.337399 12175 raft_consensus.cc:348] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.337491 12175 raft_consensus.cc:374] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.337518 12175 raft_consensus.cc:729] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.337625 12175 consensus_queue.cc:260] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.337881 12175 ts_tablet_manager.cc:1398] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:40.338163 12174 tablet_bootstrap.cc:654] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.338657 12176 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:40.338747 12176 ts_tablet_manager.cc:1367] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:40.339078 12177 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:40.339156 12176 raft_consensus.cc:348] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.339246 12176 raft_consensus.cc:374] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.339277 12176 raft_consensus.cc:729] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.339394 12176 consensus_queue.cc:260] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.339687 12176 ts_tablet_manager.cc:1398] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.339855 12176 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:40.339910 12177 tablet_bootstrap.cc:654] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.339954 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd522180bd4e53906cd9572f82ab67 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.340157 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd522180bd4e53906cd9572f82ab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.340595 12176 tablet_bootstrap.cc:654] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.341194 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 5d032fb77ac944629a45b808cc76c2f1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.341423 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d032fb77ac944629a45b808cc76c2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.341846 12175 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:40.342938 12175 tablet_bootstrap.cc:654] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.343726 12174 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:40.343813 12174 ts_tablet_manager.cc:1367] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:40.344259 12174 raft_consensus.cc:348] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.344357 12174 raft_consensus.cc:374] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.344395 12174 raft_consensus.cc:729] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.344508 12174 consensus_queue.cc:260] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.344775 12174 ts_tablet_manager.cc:1398] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:40.344991 12174 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:40.345801 12175 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:40.345873 12174 tablet_bootstrap.cc:654] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.345882 12175 ts_tablet_manager.cc:1367] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:40.346271 12175 raft_consensus.cc:348] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.346410 12175 raft_consensus.cc:374] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.346448 12175 raft_consensus.cc:729] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.346560 12175 consensus_queue.cc:260] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.346864 12175 ts_tablet_manager.cc:1398] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:40.347002 12175 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:40.347146 12177 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:40.347223 12177 ts_tablet_manager.cc:1367] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:40.347555 12177 raft_consensus.cc:348] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.347637 12177 raft_consensus.cc:374] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.347669 12177 raft_consensus.cc:729] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.347750 12175 tablet_bootstrap.cc:654] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.347784 12177 consensus_queue.cc:260] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.348048 12177 ts_tablet_manager.cc:1398] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.348837 12175 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:40.349068 12175 ts_tablet_manager.cc:1367] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:40.349517 12175 raft_consensus.cc:348] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.349596 12175 raft_consensus.cc:374] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.350059 12176 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:40.350155 12176 ts_tablet_manager.cc:1367] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:57:40.350468 12176 raft_consensus.cc:348] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.350553 12176 raft_consensus.cc:374] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.350587 12176 raft_consensus.cc:729] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.350708 12176 consensus_queue.cc:260] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.350936 12176 ts_tablet_manager.cc:1398] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:40.349628 12175 raft_consensus.cc:729] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.351145 12175 consensus_queue.cc:260] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.351413 12175 ts_tablet_manager.cc:1398] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:57:40.352319 12174 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:40.352418 12174 ts_tablet_manager.cc:1367] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:40.352866 12174 raft_consensus.cc:348] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.352991 12174 raft_consensus.cc:374] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.353034 12174 raft_consensus.cc:729] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.353144 12174 consensus_queue.cc:260] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.353405 12174 ts_tablet_manager.cc:1398] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.353622 12174 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:40.354506 12174 tablet_bootstrap.cc:654] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.355598 12174 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:40.355683 12174 ts_tablet_manager.cc:1367] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:40.356098 12174 raft_consensus.cc:348] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.356191 12174 raft_consensus.cc:374] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.356222 12174 raft_consensus.cc:729] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.356341 12174 consensus_queue.cc:260] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.356675 12174 ts_tablet_manager.cc:1398] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.363719 12282 raft_consensus.cc:482] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.363798 12282 raft_consensus.cc:504] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.364104 12282 leader_election.cc:288] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.364364 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219742bd987e4a1990233fa8c972c232" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:40.364401 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219742bd987e4a1990233fa8c972c232" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:40.364480 11245 raft_consensus.cc:2454] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.364486 11375 raft_consensus.cc:2454] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.364728 11048 leader_election.cc:302] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:40.364862 12282 raft_consensus.cc:2781] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.364925 12282 raft_consensus.cc:482] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.364960 12282 raft_consensus.cc:3037] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.365532 12282 raft_consensus.cc:504] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.365828 12282 leader_election.cc:288] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.366086 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219742bd987e4a1990233fa8c972c232" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:40.366091 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219742bd987e4a1990233fa8c972c232" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:40.366183 11375 raft_consensus.cc:3037] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.366194 11245 raft_consensus.cc:3037] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.366768 11375 raft_consensus.cc:2454] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.366901 11245 raft_consensus.cc:2454] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.367018 11050 leader_election.cc:302] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:40.367183 12282 raft_consensus.cc:2781] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.367259 12282 raft_consensus.cc:686] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:40.367383 12282 consensus_queue.cc:237] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.368686 10833 catalog_manager.cc:5484] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:40.414078 12036 raft_consensus.cc:482] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.414165 12036 raft_consensus.cc:504] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.415069 12036 leader_election.cc:288] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:40.418424 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd522180bd4e53906cd9572f82ab67" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:40.418542 10983 raft_consensus.cc:2454] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:40.418545 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd522180bd4e53906cd9572f82ab67" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:40.418661 11114 raft_consensus.cc:2454] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:40.418879 11181 leader_election.cc:302] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.419090 12036 raft_consensus.cc:2781] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.419142 12036 raft_consensus.cc:482] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.419191 12036 raft_consensus.cc:3037] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.419860 12036 raft_consensus.cc:504] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.420166 12036 leader_election.cc:288] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:40.420451 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd522180bd4e53906cd9572f82ab67" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:40.420480 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd522180bd4e53906cd9572f82ab67" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:40.420539 10983 raft_consensus.cc:3037] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.420593 11114 raft_consensus.cc:3037] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.421249 10983 raft_consensus.cc:2454] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 1.
I20240501 19:57:40.421485 11181 leader_election.cc:302] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.421638 12036 raft_consensus.cc:2781] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.421833 12036 raft_consensus.cc:686] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 19:57:40.422081 12036 consensus_queue.cc:237] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.423748 10833 catalog_manager.cc:5484] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:40.424158 12281 raft_consensus.cc:482] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.424247 12281 raft_consensus.cc:504] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.424618 12281 leader_election.cc:288] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:40.424959 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d032fb77ac944629a45b808cc76c2f1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:40.425058 11245 raft_consensus.cc:2454] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:40.425096 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d032fb77ac944629a45b808cc76c2f1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:40.425184 11113 raft_consensus.cc:2454] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:40.425341 11310 leader_election.cc:302] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 19:57:40.425554 12281 raft_consensus.cc:2781] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.425609 12281 raft_consensus.cc:482] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.425652 12281 raft_consensus.cc:3037] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.426366 12281 raft_consensus.cc:504] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.426646 12281 leader_election.cc:288] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:40.426929 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d032fb77ac944629a45b808cc76c2f1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:40.426944 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d032fb77ac944629a45b808cc76c2f1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:40.427016 11113 raft_consensus.cc:3037] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.427019 11245 raft_consensus.cc:3037] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.427589 11245 raft_consensus.cc:2454] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:40.427846 11310 leader_election.cc:302] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 19:57:40.427996 12281 raft_consensus.cc:2781] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.428090 12281 raft_consensus.cc:686] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:40.428211 12281 consensus_queue.cc:237] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.429535 10833 catalog_manager.cc:5484] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:40.437500 11114 raft_consensus.cc:2454] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 1.
I20240501 19:57:40.438050 11113 raft_consensus.cc:2454] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
W20240501 19:57:40.537396 12301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:40.537585 12301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:40.540541 12301 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:57:40.624507 11245 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.626626 11245 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.626758 11375 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.627013 11986 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.627097 11973 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.628135 11973 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.628535 11113 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.628983 11986 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.631175 10983 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.631606 12036 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:40.639987 12161 mvcc.cc:205] Tried to move back new op lower bound from 7022974814730125312 to 7022974813915041792. Current Snapshot: MvccSnapshot[applied={T|T < 7022974814714048512}]
I20240501 19:57:40.645289 11114 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:40.647943 12036 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.857 ms
time per row: 0.0499285 ms
I20240501 19:57:40.680773 10835 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:40.686198 10962 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee13ec9bb1468f90cfc184455f9fff (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.686267 11092 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee13ec9bb1468f90cfc184455f9fff (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.686470 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee13ec9bb1468f90cfc184455f9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.686471 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee13ec9bb1468f90cfc184455f9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.686946 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 318eb8fd9972435a8e51f63bac077f09 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.687161 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.687425 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 318eb8fd9972435a8e51f63bac077f09 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.687433 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 269047cc3a504d7488747166efce71fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.687685 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.687896 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269047cc3a504d7488747166efce71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.688241 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 269047cc3a504d7488747166efce71fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.688443 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269047cc3a504d7488747166efce71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.688686 12175 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:40.689181 12174 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:40.689189 11356 tablet_service.cc:1467] Processing CreateTablet for tablet 269047cc3a504d7488747166efce71fc (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.689460 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269047cc3a504d7488747166efce71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.689589 12175 tablet_bootstrap.cc:654] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.690052 12174 tablet_bootstrap.cc:654] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.690698 12175 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:40.690786 12175 ts_tablet_manager.cc:1367] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.691144 12174 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:40.691213 12174 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.691213 12175 raft_consensus.cc:348] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.691210 11355 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee13ec9bb1468f90cfc184455f9fff (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.691299 12175 raft_consensus.cc:374] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.691339 12175 raft_consensus.cc:729] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.691423 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee13ec9bb1468f90cfc184455f9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.691434 12175 consensus_queue.cc:260] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.691543 12176 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:40.691605 12174 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.691687 12175 ts_tablet_manager.cc:1398] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.691689 12174 raft_consensus.cc:374] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.691720 12174 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.691848 12175 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:40.691823 12174 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.692070 12174 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.692308 12176 tablet_bootstrap.cc:654] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.692631 12175 tablet_bootstrap.cc:654] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.693485 11354 tablet_service.cc:1467] Processing CreateTablet for tablet 318eb8fd9972435a8e51f63bac077f09 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:40.693714 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:40.693972 12176 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:40.694033 12176 ts_tablet_manager.cc:1367] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:40.694442 12176 raft_consensus.cc:348] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.694528 12176 raft_consensus.cc:374] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.694564 12176 raft_consensus.cc:729] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.694679 12176 consensus_queue.cc:260] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.694926 12176 ts_tablet_manager.cc:1398] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.695067 12176 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:40.695924 12176 tablet_bootstrap.cc:654] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.696662 12175 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:40.696732 12175 ts_tablet_manager.cc:1367] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:57:40.697027 12176 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:40.697108 12176 ts_tablet_manager.cc:1367] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.697183 12175 raft_consensus.cc:348] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.697286 12175 raft_consensus.cc:374] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.697321 12175 raft_consensus.cc:729] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.697423 12175 consensus_queue.cc:260] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.697494 12176 raft_consensus.cc:348] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.697595 12176 raft_consensus.cc:374] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.697636 12176 raft_consensus.cc:729] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.697750 12176 consensus_queue.cc:260] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.698010 12176 ts_tablet_manager.cc:1398] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:40.697875 12175 ts_tablet_manager.cc:1398] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.698159 12176 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:40.698889 12176 tablet_bootstrap.cc:654] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.700084 12176 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:40.700167 12176 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.700562 12176 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.700642 12176 raft_consensus.cc:374] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.700675 12176 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.700796 12176 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.701119 12176 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.704064 12177 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:40.704721 12177 tablet_bootstrap.cc:654] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.705616 12177 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:40.705689 12177 ts_tablet_manager.cc:1367] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.706023 12177 raft_consensus.cc:348] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.706113 12177 raft_consensus.cc:374] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.706146 12177 raft_consensus.cc:729] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.706271 12177 consensus_queue.cc:260] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.706517 12177 ts_tablet_manager.cc:1398] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:40.706657 12177 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:40.707317 12177 tablet_bootstrap.cc:654] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.708211 12177 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:40.708293 12177 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.708647 12177 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.708760 12177 raft_consensus.cc:374] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.708786 12177 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.708880 12177 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.709226 12177 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.709373 12177 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:40.710060 12177 tablet_bootstrap.cc:654] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:40.711365 12177 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:40.711447 12177 ts_tablet_manager.cc:1367] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:40.711761 12177 raft_consensus.cc:348] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.711834 12177 raft_consensus.cc:374] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:40.711867 12177 raft_consensus.cc:729] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:40.711983 12177 consensus_queue.cc:260] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.712380 12177 ts_tablet_manager.cc:1398] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:40.780786 11973 raft_consensus.cc:482] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.780874 11973 raft_consensus.cc:504] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.781198 11973 leader_election.cc:288] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.781483 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269047cc3a504d7488747166efce71fc" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:40.781550 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269047cc3a504d7488747166efce71fc" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:40.781611 10983 raft_consensus.cc:2454] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.781653 11375 raft_consensus.cc:2454] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 0.
I20240501 19:57:40.781886 11050 leader_election.cc:302] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.782123 11973 raft_consensus.cc:2781] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.782179 11973 raft_consensus.cc:482] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.782231 11973 raft_consensus.cc:3037] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.782966 11973 raft_consensus.cc:504] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.783258 11973 leader_election.cc:288] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:40.783535 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269047cc3a504d7488747166efce71fc" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:40.783583 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269047cc3a504d7488747166efce71fc" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:40.783641 10983 raft_consensus.cc:3037] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.783675 11375 raft_consensus.cc:3037] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.784317 11375 raft_consensus.cc:2454] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.784447 10983 raft_consensus.cc:2454] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 1.
I20240501 19:57:40.784648 11050 leader_election.cc:302] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:40.784889 11973 raft_consensus.cc:2781] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.784983 11973 raft_consensus.cc:686] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:40.785092 11973 consensus_queue.cc:237] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:40.786533 10835 catalog_manager.cc:5484] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:40.865180 11986 raft_consensus.cc:482] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.865258 11986 raft_consensus.cc:504] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.865512 11986 leader_election.cc:288] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:40.865774 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee13ec9bb1468f90cfc184455f9fff" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:40.865801 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee13ec9bb1468f90cfc184455f9fff" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:40.865865 10983 raft_consensus.cc:2454] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:40.865917 11114 raft_consensus.cc:2454] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:40.866112 11312 leader_election.cc:302] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.866246 11986 raft_consensus.cc:2781] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.866302 11986 raft_consensus.cc:482] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.866326 11986 raft_consensus.cc:3037] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.866959 11986 raft_consensus.cc:504] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.867229 11986 leader_election.cc:288] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:40.867475 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee13ec9bb1468f90cfc184455f9fff" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:40.867542 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee13ec9bb1468f90cfc184455f9fff" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:40.867595 10983 raft_consensus.cc:3037] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.867678 11114 raft_consensus.cc:3037] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.868458 10983 raft_consensus.cc:2454] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:40.868458 11114 raft_consensus.cc:2454] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:40.868722 11311 leader_election.cc:302] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:40.868949 11986 raft_consensus.cc:2781] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.869029 11986 raft_consensus.cc:686] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:40.869156 11986 consensus_queue.cc:237] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:40.870563 10835 catalog_manager.cc:5484] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:40.883671 12036 raft_consensus.cc:482] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:40.883752 12036 raft_consensus.cc:504] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.884114 12036 leader_election.cc:288] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:40.884627 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318eb8fd9972435a8e51f63bac077f09" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:40.884729 10983 raft_consensus.cc:2454] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:40.884974 11181 leader_election.cc:302] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.885123 12036 raft_consensus.cc:2781] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:40.885169 12036 raft_consensus.cc:482] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:40.885197 12036 raft_consensus.cc:3037] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.885828 12036 raft_consensus.cc:504] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.886107 12036 leader_election.cc:288] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:40.886430 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318eb8fd9972435a8e51f63bac077f09" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:40.886549 10983 raft_consensus.cc:3037] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:40.887260 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318eb8fd9972435a8e51f63bac077f09" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:40.887339 10983 raft_consensus.cc:2454] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 1.
I20240501 19:57:40.887339 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318eb8fd9972435a8e51f63bac077f09" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:40.887372 11375 raft_consensus.cc:2454] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:40.887578 11181 leader_election.cc:302] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:40.887826 12036 raft_consensus.cc:2781] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:40.887912 12036 raft_consensus.cc:686] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 19:57:40.888036 12036 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:40.889405 10835 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
W20240501 19:57:41.018828 12352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:41.019038 12352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:41.021921 12352 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:57:41.091099 10983 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.091656 12036 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.092449 11375 raft_consensus.cc:3037] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.096037 11375 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.097136 12298 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.097190 11114 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.097913 11986 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.102200 11376 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.102567 12282 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.102564 10983 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.102998 11986 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.105295 10982 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.107394 12282 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.69 ms
time per row: 0.050845 ms
I20240501 19:57:41.161949 10832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:41.166975 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 25ea8d7ab1834abfa34ff2357aa554b0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.167258 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.167282 11354 tablet_service.cc:1467] Processing CreateTablet for tablet ee298406a4824f1aa2fa742b7dec1026 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.167513 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.167773 10963 tablet_service.cc:1467] Processing CreateTablet for tablet ee298406a4824f1aa2fa742b7dec1026 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.167973 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.167922 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2db0f85ac4a90a1df779f89398463 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.168130 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d2db0f85ac4a90a1df779f89398463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.169406 12176 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:41.169514 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 25ea8d7ab1834abfa34ff2357aa554b0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.169736 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.170187 12176 tablet_bootstrap.cc:654] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.170265 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2db0f85ac4a90a1df779f89398463 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.170512 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d2db0f85ac4a90a1df779f89398463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.171507 12176 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:41.171600 12176 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:41.171826 12174 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:41.171906 12177 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:41.172025 12176 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.172135 12176 raft_consensus.cc:374] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.172168 12176 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.172276 12176 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.170265 11224 tablet_service.cc:1467] Processing CreateTablet for tablet ee298406a4824f1aa2fa742b7dec1026 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.172535 12176 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.172644 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.172669 12174 tablet_bootstrap.cc:654] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.172686 12176 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:41.172776 12177 tablet_bootstrap.cc:654] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.173843 12176 tablet_bootstrap.cc:654] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.173949 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 25ea8d7ab1834abfa34ff2357aa554b0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.174203 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.175155 12176 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:41.175220 12176 ts_tablet_manager.cc:1367] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:41.175415 12177 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:41.175487 12177 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:41.175654 12176 raft_consensus.cc:348] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.175731 12176 raft_consensus.cc:374] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.175786 12176 raft_consensus.cc:729] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.175904 12176 consensus_queue.cc:260] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.175930 12177 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.176007 12177 raft_consensus.cc:374] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.176040 12177 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.176144 12177 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.176189 12176 ts_tablet_manager.cc:1398] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:41.176277 12175 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:41.176393 12177 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.176525 12177 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:41.174151 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 54d2db0f85ac4a90a1df779f89398463 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.177166 12175 tablet_bootstrap.cc:654] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.177400 12177 tablet_bootstrap.cc:654] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.177496 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d2db0f85ac4a90a1df779f89398463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.178203 12175 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:41.178279 12175 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:41.178665 12175 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.178750 12175 raft_consensus.cc:374] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.178776 12175 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.178867 12175 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.179142 12175 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.179278 12175 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:41.179811 12174 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:41.179889 12174 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:41.180074 12175 tablet_bootstrap.cc:654] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.180300 12174 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.180398 12174 raft_consensus.cc:374] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.180430 12174 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.180541 12174 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.180819 12174 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:41.180996 12174 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:41.181636 12174 tablet_bootstrap.cc:654] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.182724 12174 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:41.182821 12174 ts_tablet_manager.cc:1367] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:41.182901 12177 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:41.182978 12177 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:41.183283 12174 raft_consensus.cc:348] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.183334 12177 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.183384 12174 raft_consensus.cc:374] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.183414 12177 raft_consensus.cc:374] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.183420 12174 raft_consensus.cc:729] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.183462 12177 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.183558 12174 consensus_queue.cc:260] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.183596 12177 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.183830 12177 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:41.183864 12174 ts_tablet_manager.cc:1398] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.184007 12174 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:41.184818 12174 tablet_bootstrap.cc:654] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.184885 12175 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:41.185014 12175 ts_tablet_manager.cc:1367] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:41.185436 12175 raft_consensus.cc:348] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.185534 12175 raft_consensus.cc:374] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.185567 12175 raft_consensus.cc:729] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.185680 12175 consensus_queue.cc:260] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.185865 12174 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:41.185931 12174 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:41.185940 12175 ts_tablet_manager.cc:1398] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:41.186333 12174 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.186417 12174 raft_consensus.cc:374] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.186448 12174 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.186568 12174 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.186832 12174 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.191679 11986 raft_consensus.cc:482] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.191754 11986 raft_consensus.cc:504] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.192147 11986 leader_election.cc:288] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.192404 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:41.192469 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:41.192520 10982 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:41.192575 11244 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:41.192786 11312 leader_election.cc:302] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.192986 11986 raft_consensus.cc:2781] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.193051 11986 raft_consensus.cc:482] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.193100 11986 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.193897 11986 raft_consensus.cc:504] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.194164 11986 leader_election.cc:288] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.194463 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:41.194485 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:41.194546 10982 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.194581 11244 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.195142 10982 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:41.195223 11244 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:41.195390 11312 leader_election.cc:302] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.195590 11986 raft_consensus.cc:2781] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.195711 11986 raft_consensus.cc:686] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:41.195848 11986 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.197191 10841 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:41.270851 12036 raft_consensus.cc:482] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.270946 12036 raft_consensus.cc:504] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.271271 12036 leader_election.cc:288] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.271545 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2db0f85ac4a90a1df779f89398463" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:41.271553 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2db0f85ac4a90a1df779f89398463" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:41.271663 11376 raft_consensus.cc:2454] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:41.271661 11114 raft_consensus.cc:2454] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 0.
I20240501 19:57:41.271904 11180 leader_election.cc:302] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:41.272071 12036 raft_consensus.cc:2781] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.272146 12036 raft_consensus.cc:482] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.272179 12036 raft_consensus.cc:3037] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.273022 12036 raft_consensus.cc:504] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.273330 12036 leader_election.cc:288] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.273679 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2db0f85ac4a90a1df779f89398463" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:41.273761 11376 raft_consensus.cc:3037] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.273734 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54d2db0f85ac4a90a1df779f89398463" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:41.273833 11114 raft_consensus.cc:3037] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.274488 11114 raft_consensus.cc:2454] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 1.
I20240501 19:57:41.274581 11376 raft_consensus.cc:2454] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 1.
I20240501 19:57:41.274878 11180 leader_election.cc:302] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:41.275085 12036 raft_consensus.cc:2781] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.275156 12036 raft_consensus.cc:686] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 19:57:41.275262 12036 consensus_queue.cc:237] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.281405 10832 catalog_manager.cc:5484] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:41.350258 12386 raft_consensus.cc:482] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.350365 12386 raft_consensus.cc:504] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.350733 12386 leader_election.cc:288] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.350994 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ea8d7ab1834abfa34ff2357aa554b0" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:41.351065 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ea8d7ab1834abfa34ff2357aa554b0" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:41.351105 11114 raft_consensus.cc:2454] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.351194 11244 raft_consensus.cc:2454] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.351377 10918 leader_election.cc:302] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.351521 12386 raft_consensus.cc:2781] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.351575 12386 raft_consensus.cc:482] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.351619 12386 raft_consensus.cc:3037] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.352351 12386 raft_consensus.cc:504] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.352632 12386 leader_election.cc:288] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.352931 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ea8d7ab1834abfa34ff2357aa554b0" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:41.352938 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ea8d7ab1834abfa34ff2357aa554b0" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:41.353029 11114 raft_consensus.cc:3037] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.353029 11244 raft_consensus.cc:3037] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.353835 11114 raft_consensus.cc:2454] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.353835 11244 raft_consensus.cc:2454] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.354086 10917 leader_election.cc:302] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.354224 12386 raft_consensus.cc:2781] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.354301 12386 raft_consensus.cc:686] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:41.354415 12386 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:41.355909 10832 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
W20240501 19:57:41.458803 12404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:41.458976 12404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:41.461796 12404 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 19:57:41.528280 10982 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.528280 11244 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.528666 12281 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.528841 11986 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.530586 11376 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.531399 11114 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.531833 12036 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.532321 12036 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.558573 11244 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.558808 11114 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.559041 12386 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.559417 12386 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.9239 ms
time per row: 0.047962 ms
I20240501 19:57:41.599232 10835 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:41.604187 10963 tablet_service.cc:1467] Processing CreateTablet for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.604449 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.604565 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd958b083bb430a99130a18f91acdf7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.604768 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.604755 11092 tablet_service.cc:1467] Processing CreateTablet for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.604988 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.606182 12177 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:41.606660 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd958b083bb430a99130a18f91acdf7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.606866 12175 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:41.606889 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.606947 12177 tablet_bootstrap.cc:654] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.607746 12175 tablet_bootstrap.cc:654] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.608130 11355 tablet_service.cc:1467] Processing CreateTablet for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.608383 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.608433 11224 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd958b083bb430a99130a18f91acdf7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.608662 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.608762 12177 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:41.608827 12177 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:41.609129 11354 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6621e1ff294dc9a474e076873cc72c (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.609200 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6621e1ff294dc9a474e076873cc72c (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.609313 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6621e1ff294dc9a474e076873cc72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.609418 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6621e1ff294dc9a474e076873cc72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.609534 12177 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.609619 12177 raft_consensus.cc:374] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.609658 12177 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.609782 12177 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.610016 12177 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.610154 12177 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:41.610612 12174 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:41.610983 12177 tablet_bootstrap.cc:654] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.611214 12176 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:41.611356 12174 tablet_bootstrap.cc:654] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.612056 12176 tablet_bootstrap.cc:654] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.612233 12175 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:41.612298 12175 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:57:41.612591 12175 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.612677 12175 raft_consensus.cc:374] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.612717 12175 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.612835 12175 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.613158 12176 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:41.613158 12175 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.613233 12176 ts_tablet_manager.cc:1367] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:41.613314 12175 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:41.613570 12176 raft_consensus.cc:348] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.613682 12176 raft_consensus.cc:374] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.613711 12176 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.613826 12176 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.614066 12176 ts_tablet_manager.cc:1398] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.614301 12174 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:41.614370 12174 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:41.614362 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6621e1ff294dc9a474e076873cc72c (DEFAULT_TABLE table=rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:41.614565 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6621e1ff294dc9a474e076873cc72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:41.614651 12175 tablet_bootstrap.cc:654] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.614782 12174 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.614863 12174 raft_consensus.cc:374] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.614889 12174 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.614992 12174 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.615263 12174 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:41.615696 12175 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:41.615758 12175 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:41.616078 12174 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:41.616106 12175 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.616196 12175 raft_consensus.cc:374] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.616305 12175 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.616410 12175 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.616748 12175 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.616909 12174 tablet_bootstrap.cc:654] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.619787 12176 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:41.620517 12176 tablet_bootstrap.cc:654] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.620810 12174 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:41.620880 12174 ts_tablet_manager.cc:1367] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:41.621031 12177 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:41.621111 12177 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:57:41.621343 12174 raft_consensus.cc:348] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.621443 12174 raft_consensus.cc:374] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.621423 12177 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.621474 12174 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.621488 12176 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:41.621516 12177 raft_consensus.cc:374] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.621555 12176 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:41.621553 12177 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.621605 12174 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.621671 12177 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.621871 12174 ts_tablet_manager.cc:1398] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:41.621963 12177 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:41.621960 12176 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.622041 12176 raft_consensus.cc:374] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.622074 12176 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.622119 12177 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:41.622174 12176 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.622421 12176 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:41.622850 12177 tablet_bootstrap.cc:654] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:41.623823 12177 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:41.623888 12177 ts_tablet_manager.cc:1367] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:41.624279 12177 raft_consensus.cc:348] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.625000 12177 raft_consensus.cc:374] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:41.625042 12177 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.625135 12177 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.626017 12177 ts_tablet_manager.cc:1398] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:41.716723 12386 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.716802 12386 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.717186 12386 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.717478 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:41.717499 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:41.717602 11376 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.717612 11114 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.717867 10919 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.718031 12386 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.718094 12386 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.718123 12386 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.718806 12386 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.719115 12386 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.719388 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:41.719385 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:41.719492 11114 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.719492 11376 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.720328 11114 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.720328 11376 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.720604 10919 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.720764 12386 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.720875 12386 raft_consensus.cc:686] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:41.721035 12386 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:41.722640 10832 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:41.757351 12386 raft_consensus.cc:482] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.757439 12386 raft_consensus.cc:504] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.757745 12386 leader_election.cc:288] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.758069 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:41.758069 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:41.758183 11114 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.758183 11245 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.758424 10917 leader_election.cc:302] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.758577 12386 raft_consensus.cc:2781] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.758633 12386 raft_consensus.cc:482] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.758664 12386 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.759289 12386 raft_consensus.cc:504] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.759544 12386 leader_election.cc:288] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:41.759806 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:41.759806 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:41.759905 11114 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.759905 11245 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.760694 11114 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.760694 11245 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.761022 10918 leader_election.cc:302] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.761199 12386 raft_consensus.cc:2781] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.761312 12386 raft_consensus.cc:686] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:41.761444 12386 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.762967 10832 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:41.782994 12386 raft_consensus.cc:482] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.783114 12386 raft_consensus.cc:504] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.783428 12386 leader_election.cc:288] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.783737 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:41.783759 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 19:57:41.783835 11245 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.783862 11376 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 0.
I20240501 19:57:41.784070 10919 leader_election.cc:302] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.784214 12386 raft_consensus.cc:2781] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:41.784296 12386 raft_consensus.cc:482] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:41.784328 12386 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.785156 12386 raft_consensus.cc:504] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.785665 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:41.785705 12386 leader_election.cc:288] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Requested vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:41.785766 11245 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.786379 11245 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.786679 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:41.786746 10917 leader_election.cc:302] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:41.786780 11376 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:41.786897 12386 raft_consensus.cc:2781] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:41.787190 12386 raft_consensus.cc:686] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:41.787298 12386 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:41.787495 11376 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 1.
I20240501 19:57:41.788712 10832 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
W20240501 19:57:41.892998 12455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:41.893216 12455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:41.895890 12455 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:57:41.976203 11245 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.976440 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.976470 11114 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.976940 11376 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.977066 12386 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.977128 12232 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.977329 12232 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.979138 12386 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.985813 11375 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.986176 12232 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:41.989042 11242 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:41.993129 12386 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.527 ms
time per row: 0.0532637 ms
I20240501 19:57:42.038589 10838 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40216:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:57:42.043416 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.043435 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.043475 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.043689 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.043691 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.043694 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.045258 12175 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:42.045611 12176 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:42.045650 12174 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:42.046070 12175 tablet_bootstrap.cc:654] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.046439 12176 tablet_bootstrap.cc:654] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.046523 12174 tablet_bootstrap.cc:654] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.046561 10963 tablet_service.cc:1467] Processing CreateTablet for tablet 2def1cda07c349c69b3abe25804d3d1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.046799 10963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.047361 11355 tablet_service.cc:1467] Processing CreateTablet for tablet 156d2ba7ff2c403db2f63cbae4acfec1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.047499 11354 tablet_service.cc:1467] Processing CreateTablet for tablet 2def1cda07c349c69b3abe25804d3d1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.047560 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.047667 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.048130 11094 tablet_service.cc:1467] Processing CreateTablet for tablet 156d2ba7ff2c403db2f63cbae4acfec1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.048336 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.049136 11225 tablet_service.cc:1467] Processing CreateTablet for tablet 156d2ba7ff2c403db2f63cbae4acfec1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.049357 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.049687 12177 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:42.049682 12175 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:42.049758 12175 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:57:42.050136 12175 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.050227 12175 raft_consensus.cc:374] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.050294 12175 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.050410 12175 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.050474 12177 tablet_bootstrap.cc:654] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.050715 12175 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.050868 12175 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:42.051169 11092 tablet_service.cc:1467] Processing CreateTablet for tablet 2def1cda07c349c69b3abe25804d3d1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:42.051371 11092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:42.051628 12175 tablet_bootstrap.cc:654] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.052639 12175 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:42.052703 12175 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:42.053140 12175 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.053305 12175 raft_consensus.cc:374] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.053362 12175 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.053470 12175 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.053745 12175 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.053864 12176 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:42.053917 12175 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:42.053962 12176 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:42.054517 12176 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.054616 12176 raft_consensus.cc:374] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.054654 12176 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.054704 12175 tablet_bootstrap.cc:654] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.054770 12176 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.055035 12176 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.055476 12177 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8: No bootstrap required, opened a new log
I20240501 19:57:42.055557 12177 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:42.055951 12175 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: No bootstrap required, opened a new log
I20240501 19:57:42.055960 12177 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.056018 12175 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:42.056068 12177 raft_consensus.cc:374] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.056111 12177 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.056255 12177 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.056417 12175 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.056514 12175 raft_consensus.cc:374] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.056526 12177 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:42.056546 12175 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.056782 12176 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:42.056842 12175 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.057615 12176 tablet_bootstrap.cc:654] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.058270 12175 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:42.058619 12174 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:42.058696 12174 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 19:57:42.059135 12174 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.059222 12174 raft_consensus.cc:374] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.059258 12174 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.059376 12174 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.059624 12174 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.059763 12174 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:42.060570 12174 tablet_bootstrap.cc:654] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.061518 12174 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166: No bootstrap required, opened a new log
I20240501 19:57:42.061590 12174 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:42.061669 12176 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:42.061736 12176 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:57:42.062034 12174 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.062110 12174 raft_consensus.cc:374] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.062144 12174 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.062116 12176 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.062250 12176 raft_consensus.cc:374] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.062286 12176 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.062287 12174 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.062397 12176 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.062587 12174 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.062649 12176 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.062790 12176 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:42.063580 12176 tablet_bootstrap.cc:654] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:42.064419 12176 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: No bootstrap required, opened a new log
I20240501 19:57:42.064491 12176 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:42.064852 12176 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.064965 12176 raft_consensus.cc:374] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:42.065003 12176 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:42.065110 12176 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.065321 12176 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:42.084483 12281 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:42.084556 12281 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.084839 12281 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:42.085058 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:42.085175 11114 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.085151 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:57:42.085265 10983 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.085412 11311 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.085553 12281 raft_consensus.cc:2781] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:42.085598 12281 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:42.085623 12281 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.086251 12281 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.086519 12281 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:42.086788 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:42.086831 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:42.086887 11114 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.086910 10983 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.087529 11114 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.087566 10983 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.087788 11311 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.087980 12281 raft_consensus.cc:2781] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:42.088057 12281 raft_consensus.cc:686] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:42.088207 12281 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.089586 10839 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:42.098665 12281 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:42.098737 12281 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.099004 12281 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:42.099247 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:42.099345 11114 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.099318 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:42.099431 11245 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.099602 11311 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.099740 12281 raft_consensus.cc:2781] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:42.099792 12281 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:42.099828 12281 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.100416 12281 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.100692 12281 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:42.100951 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:42.101002 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:42.101047 11114 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.101081 11245 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.101655 11114 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.101724 11245 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.101986 11311 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.102150 12281 raft_consensus.cc:2781] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:42.102248 12281 raft_consensus.cc:686] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:42.102358 12281 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:42.103740 10839 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:42.330857 12281 raft_consensus.cc:482] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:42.330945 12281 raft_consensus.cc:504] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.331259 12281 leader_election.cc:288] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:42.331528 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:57:42.331533 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 19:57:42.331635 11245 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.331640 11114 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 0.
I20240501 19:57:42.331916 11311 leader_election.cc:302] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.332056 12281 raft_consensus.cc:2781] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:42.332125 12281 raft_consensus.cc:482] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:42.332152 12281 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.332888 12281 raft_consensus.cc:504] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.333187 12281 leader_election.cc:288] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:42.333487 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:42.333487 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:42.333596 11114 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.333596 11245 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:42.334470 11114 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.334481 11245 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 1.
I20240501 19:57:42.334733 11311 leader_election.cc:302] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:42.334872 12281 raft_consensus.cc:2781] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:42.334956 12281 raft_consensus.cc:686] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:42.335083 12281 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:42.336483 10839 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
W20240501 19:57:42.438831 12506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:42.439026 12506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:42.441905 12506 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:57:42.516685 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.517592 11114 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.517724 11113 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.517845 11112 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.517931 11986 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.518529 11245 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.518788 11242 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:42.520331 12538 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.520646 12538 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.520819 11986 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.521016 12538 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.521683 12281 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:42.529729 12533 mvcc.cc:205] Tried to move back new op lower bound from 7022974822480941056 to 7022974821725450240. Current Snapshot: MvccSnapshot[applied={T|T < 7022974822466801664}]
INSERT report
rows total: 2000
time total: 96.2103 ms
time per row: 0.0481052 ms
I20240501 19:57:42.577847 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.69:38171
--local_ip_for_outbound_sockets=127.30.1.69
--tserver_master_addrs=127.30.1.126:33217
--webserver_port=33209
--webserver_interface=127.30.1.69
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 19:57:42.676461 12559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:42.676620 12559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:42.676658 12559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:42.679363 12559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:42.679419 12559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:42.679487 12559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.69
I20240501 19:57:42.682782 12559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.69:38171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.1.69
--webserver_port=33209
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.12559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.69
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:42.683421 12559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:42.684284 12559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:42.691817 12565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:42.692296 12564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:42.693050 12567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:42.693847 12559 server_base.cc:1034] running on GCE node
I20240501 19:57:42.694142 12559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:42.694617 12559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:42.695749 12559 hybrid_clock.cc:648] HybridClock initialized: now 1714593462695729 us; error 33 us; skew 500 ppm
I20240501 19:57:42.697371 12559 webserver.cc:460] Webserver started at http://127.30.1.69:33209/ using document root <none> and password file <none>
I20240501 19:57:42.697825 12559 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:42.697899 12559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:42.700740 12559 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:42.702368 12573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.703271 12559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:42.703362 12559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "417972166f4d40ca9b4b6471a90be6af"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:42.703687 12559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:42.777223 12559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:42.777712 12559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:42.777844 12559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:42.778328 12559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:42.779145 12559 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:42.779193 12559 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.779233 12559 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:42.779263 12559 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.788292 12559 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.69:38171
I20240501 19:57:42.788337 12686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.69:38171 every 8 connection(s)
I20240501 19:57:42.789311 12559 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:57:42.794858 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 12559
I20240501 19:57:42.795289 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.70:35165
--local_ip_for_outbound_sockets=127.30.1.70
--tserver_master_addrs=127.30.1.126:33217
--webserver_port=41109
--webserver_interface=127.30.1.70
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:42.799086 12687 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:42.799257 12687 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:42.799630 12687 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:42.800616 10839 ts_manager.cc:194] Re-registered known tserver with Master: 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:42.801450 10839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.69:41741
W20240501 19:57:42.897269 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:42.897428 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:42.897465 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:42.900439 12690 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:42.900509 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:42.900599 12690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.70
I20240501 19:57:42.904253 12690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.70:35165
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.30.1.70
--webserver_port=41109
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.12690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.70
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:42.904949 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:42.905833 12690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:42.913163 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:42.913931 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:42.914487 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:42.915163 12690 server_base.cc:1034] running on GCE node
I20240501 19:57:42.915431 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:42.915803 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:42.916967 12690 hybrid_clock.cc:648] HybridClock initialized: now 1714593462916950 us; error 24 us; skew 500 ppm
I20240501 19:57:42.918540 12690 webserver.cc:460] Webserver started at http://127.30.1.70:41109/ using document root <none> and password file <none>
I20240501 19:57:42.918979 12690 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:42.919049 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:42.921831 12690 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:42.923462 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.924295 12690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:42.924376 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "14002a0a4c674b438c78ca5db73637cc"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:42.924759 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:42.936152 12690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:42.936638 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:42.936889 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:42.937395 12690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:42.938102 12690 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:42.938144 12690 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.938177 12690 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:42.938201 12690 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:42.945938 12690 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.70:35165
I20240501 19:57:42.945984 12818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.70:35165 every 8 connection(s)
I20240501 19:57:42.946775 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:57:42.952613 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 12690
I20240501 19:57:42.953095 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.71:32843
--local_ip_for_outbound_sockets=127.30.1.71
--tserver_master_addrs=127.30.1.126:33217
--webserver_port=34171
--webserver_interface=127.30.1.71
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:42.955173 12819 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:42.955369 12819 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:42.955725 12819 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:42.964442 10839 ts_manager.cc:194] Re-registered known tserver with Master: 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 19:57:42.965124 10839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.70:46045
W20240501 19:57:43.057971 12822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.058149 12822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:43.058213 12822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.061102 12822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:43.061184 12822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:43.061293 12822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.71
I20240501 19:57:43.064867 12822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.71:32843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.30.1.71
--webserver_port=34171
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.12822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.71
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:43.065667 12822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:43.066653 12822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:43.074647 12827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:43.075178 12830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:43.075901 12828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:43.076660 12822 server_base.cc:1034] running on GCE node
I20240501 19:57:43.077072 12822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:43.077602 12822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:43.078778 12822 hybrid_clock.cc:648] HybridClock initialized: now 1714593463078752 us; error 36 us; skew 500 ppm
I20240501 19:57:43.080709 12822 webserver.cc:460] Webserver started at http://127.30.1.71:34171/ using document root <none> and password file <none>
I20240501 19:57:43.081287 12822 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:43.081365 12822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:43.084674 12822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:43.086628 12837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.087477 12822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:43.087576 12822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ea26a64181784350bcd3540d0633b219"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:43.087945 12822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:43.106562 12822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:43.107046 12822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:43.107183 12822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:43.107635 12822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:43.108474 12822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:43.108528 12822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.108582 12822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:43.108608 12822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.117022 12822 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.71:32843
I20240501 19:57:43.117061 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.71:32843 every 8 connection(s)
I20240501 19:57:43.117911 12822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:57:43.119360 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 12822
I20240501 19:57:43.119767 30725 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
/tmp/dist-test-tasksjaB1H/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.1.72:35171
--local_ip_for_outbound_sockets=127.30.1.72
--tserver_master_addrs=127.30.1.126:33217
--webserver_port=39617
--webserver_interface=127.30.1.72
--builtin_ntp_servers=127.30.1.84:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 19:57:43.126958 12951 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:43.127167 12951 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:43.127648 12951 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:43.128600 10839 ts_manager.cc:194] Re-registered known tserver with Master: ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843)
I20240501 19:57:43.129184 10839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.71:50121
W20240501 19:57:43.220813 12954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.221014 12954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:43.221052 12954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.224022 12954 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 19:57:43.224088 12954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:43.224144 12954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.1.72
I20240501 19:57:43.227311 12954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.1.84:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.1.72:35171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.30.1.72
--webserver_port=39617
--tserver_master_addrs=127.30.1.126:33217
--never_fsync=true
--heap_profile_path=/tmp/kudu.12954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.1.72
--log_dir=/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:43.227986 12954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:43.228899 12954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:43.236358 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:43.236850 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:43.237433 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:43.238255 12954 server_base.cc:1034] running on GCE node
I20240501 19:57:43.238564 12954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:43.239048 12954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:43.240207 12954 hybrid_clock.cc:648] HybridClock initialized: now 1714593463240198 us; error 24 us; skew 500 ppm
I20240501 19:57:43.246439 12954 webserver.cc:460] Webserver started at http://127.30.1.72:39617/ using document root <none> and password file <none>
I20240501 19:57:43.246912 12954 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:43.246984 12954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:43.249879 12954 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:43.251626 12969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.252594 12954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:43.252696 12954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
format_stamp: "Formatted at 2024-05-01 19:57:33 on dist-test-slave-htmr"
I20240501 19:57:43.253083 12954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:43.286274 12954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:43.286751 12954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:43.286908 12954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:43.287376 12954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:43.288339 12954 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:43.288399 12954 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.288445 12954 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:43.288476 12954 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:43.297438 12954 rpc_server.cc:307] RPC server started. Bound to: 127.30.1.72:35171
I20240501 19:57:43.297477 13082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.1.72:35171 every 8 connection(s)
I20240501 19:57:43.298362 12954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:57:43.305362 30725 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu as pid 12954
I20240501 19:57:43.307081 13083 heartbeater.cc:347] Connected to a master server at 127.30.1.126:33217
I20240501 19:57:43.307303 13083 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:43.307804 13083 heartbeater.cc:511] Master 127.30.1.126:33217 requested a full tablet report, sending...
I20240501 19:57:43.308794 10839 ts_manager.cc:194] Re-registered known tserver with Master: e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:43.309501 10839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.1.72:44359
I20240501 19:57:43.674387 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:43.677975 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.680289 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:43.683148 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:43.690348 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.691120 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:43.709259 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.713914 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:43.721417 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:43.732750 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.739861 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:43.744812 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:43.745059 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:43.749487 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:43.767089 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.767913 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.777109 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.785161 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.794107 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.798518 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.799513 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.803117 12687 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:43.815680 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:43.831373 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.834508 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:43.919420 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.922268 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:43.929546 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:43.930261 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.932588 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:43.934216 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:43.967413 12819 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:43.973042 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:43.982475 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:43.984194 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:43.985069 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:43.986552 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:43.994139 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.005295 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.025657 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.027397 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.029711 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.131253 12951 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:44.147150 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.157861 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.162015 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.166290 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.179060 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.179296 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.179397 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.193619 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.203534 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.203954 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.209401 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.217804 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.229051 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.251478 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.253562 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.264321 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.266855 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.285439 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.292837 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.296205 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.297163 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.310667 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.311307 13083 heartbeater.cc:503] Master 127.30.1.126:33217 was elected leader, sending a full tablet report...
I20240501 19:57:44.312362 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.320039 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.403137 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.413170 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.419474 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.436903 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.442643 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.443650 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.444254 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.461632 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.472376 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.474345 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.474756 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.485997 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.498320 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.506933 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.507537 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.523643 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.587324 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.590008 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.596428 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.603420 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.607128 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.623008 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.640945 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.647239 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.666208 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.666285 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.669466 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.674768 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.689800 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.703409 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.724426 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.726070 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.730891 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.736550 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.752643 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.759722 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.761837 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.774390 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.777161 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.779899 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.866001 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.866955 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.867187 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.871064 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.890779 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.894977 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.896215 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.897058 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.912397 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:44.920728 11112 consensus_queue.cc:237] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:44.922849 11375 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:44.922863 10982 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:44.923485 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:44.923674 13530 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:44.923986 13530 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:44.925237 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.926437 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:57:44.947863 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 269047cc3a504d7488747166efce71fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:44.950919 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:44.959168 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.965658 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:44.972666 13843 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:44.973127 11375 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:44.974202 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:44.975971 10836 catalog_manager.cc:5484] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:44.976423 10982 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:44.998559 10982 consensus_queue.cc:237] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.000226 11375 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.000317 11242 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.000692 12491 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.001051 12491 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.006458 13865 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.007581 11375 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.009903 10838 catalog_manager.cc:5484] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
W20240501 19:57:45.017576 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e48d711c5714ee58ebbcbd094ebf3e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.018409 11242 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.054131 10982 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.054322 10983 consensus_queue.cc:237] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.055680 11112 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.056169 11375 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.057641 13801 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.057744 11112 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.058146 13801 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.059054 13865 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.059155 11376 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.060103 11375 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.062745 11242 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.063473 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbc6ed9f4bf34b1caa462aa9d85c7a16. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.063568 12535 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.064188 11113 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.064388 13865 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.065006 12535 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.065063 10917 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a1bbe74dbc74fa687b0a64d3aa203f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.066783 12535 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.068573 13801 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.068588 11242 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.069552 11375 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.070663 10838 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.071236 13865 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.071717 11112 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.072641 10832 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.073280 11112 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.073750 11375 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.075737 11112 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.084821 10838 catalog_manager.cc:5484] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.087085 10983 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
W20240501 19:57:45.089193 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25ea8d7ab1834abfa34ff2357aa554b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.089203 11242 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.089291 11112 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.089637 13953 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.090446 11311 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 156d2ba7ff2c403db2f63cbae4acfec1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.090636 13952 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.094857 13952 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.096477 11113 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.099424 10834 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.100973 11242 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.121098 11375 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.124785 11112 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.125216 13941 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.126240 11242 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.126703 12535 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.129868 11310 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95b8c1a34cc94d29986289939b2159bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.138269 12535 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.138549 11112 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.140344 11242 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.140525 11112 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.140938 10835 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
W20240501 19:57:45.142661 11049 consensus_peers.cc:485] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c49b9c5ebaab42e18fe7bbe8127d569e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.142896 11242 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.143265 13850 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.143302 10983 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.146381 13850 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.152333 11375 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
W20240501 19:57:45.154132 11311 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a170766f497f491ba9b45f92f2a0e751. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.154448 10982 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.155035 11242 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.155416 12535 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.155936 12535 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.158573 13850 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.161229 10983 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.161629 10839 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.163466 11245 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.166496 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.169996 12535 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.172457 10838 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.173573 11242 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.175141 10982 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.176980 11375 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.178442 11112 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.179225 11311 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2def1cda07c349c69b3abe25804d3d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.179414 13941 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.181273 10982 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.182489 12535 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.186233 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:45.187955 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:45.189914 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.194633 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.198133 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:45.198436 14033 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.199440 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.202138 10835 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.203385 11112 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.208148 10983 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.221168 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:45.265547 10982 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.267144 11242 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.267773 11112 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.271483 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fcd6776aebf47c99f05850de83b36e6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.271782 13865 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.273222 13865 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.290238 13865 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.291380 11375 consensus_queue.cc:237] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.292681 11242 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.293262 11112 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.294663 10841 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.298553 10982 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.299418 12832 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.300904 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 19:57:45.303109 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3ee13ec9bb1468f90cfc184455f9fff. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.303360 11112 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.303839 12832 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.309278 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.317334 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:45.320091 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.322485 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:45.325469 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:45.334336 12832 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.337221 10836 catalog_manager.cc:5484] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.339087 10982 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.341646 11112 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.344626 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:45.355548 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.381175 10982 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.382598 11242 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.382968 13801 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.384193 11112 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.385067 12491 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.386911 10918 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd958b083bb430a99130a18f91acdf7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.389991 12491 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.392289 11242 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.393488 10841 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.398754 11112 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.422129 11375 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.423646 10982 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.423753 11112 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.424150 14033 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.424329 12535 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.424518 11311 consensus_peers.cc:485] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19bfb955b9c94fdb8e0381e5b1855248. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.433243 11242 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.435178 10983 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.435603 11375 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.436517 12441 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.436925 12441 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.438628 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.442533 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 19:57:45.443818 11179 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 318eb8fd9972435a8e51f63bac077f09. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.444773 12535 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.446919 11112 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.447269 10834 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.449004 10983 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.452210 14253 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.453527 11375 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.455942 10837 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.456156 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:45.457798 10982 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.477392 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:45.478976 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.480774 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:45.488111 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.498548 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.517144 14291 ts_tablet_manager.cc:902] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:45.517880 14291 tablet_copy_client.cc:308] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:45.529161 14327 ts_tablet_manager.cc:902] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:45.530033 14327 tablet_copy_client.cc:308] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:45.531235 11376 consensus_queue.cc:237] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.533108 11112 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:57:45.533231 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3ee13ec9bb1468f90cfc184455f9fff. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.533535 10983 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:45.534350 13941 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:45.535251 11003 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:50311)
W20240501 19:57:45.535291 11310 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3ee13ec9bb1468f90cfc184455f9fff. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.535374 11003 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:50311: session id = 14002a0a4c674b438c78ca5db73637cc-0a1bbe74dbc74fa687b0a64d3aa203f3
I20240501 19:57:45.535508 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 269047cc3a504d7488747166efce71fc from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:53481)
I20240501 19:57:45.535609 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 269047cc3a504d7488747166efce71fc from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:53481: session id = ea26a64181784350bcd3540d0633b219-269047cc3a504d7488747166efce71fc
I20240501 19:57:45.535669 12832 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:45.536780 11003 tablet_copy_source_session.cc:205] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.536825 11134 tablet_copy_source_session.cc:205] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.538774 12832 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.540359 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269047cc3a504d7488747166efce71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.541484 11112 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.541613 10839 catalog_manager.cc:5484] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.543542 10983 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.553958 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.560431 14291 tablet_copy_client.cc:783] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.560731 14291 tablet_copy_client.cc:647] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.561179 14327 tablet_copy_client.cc:783] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.561398 14327 tablet_copy_client.cc:647] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.564950 14327 tablet_copy_client.cc:523] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.564975 14291 tablet_copy_client.cc:523] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.566481 14291 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:45.566481 14327 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:45.573032 14327 log.cc:824] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:45.575838 14339 ts_tablet_manager.cc:902] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:45.576933 14339 tablet_copy_client.cc:308] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:45.578109 14340 ts_tablet_manager.cc:902] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:45.578598 14340 tablet_copy_client.cc:308] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:45.579506 14291 log.cc:824] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:45.583089 14347 ts_tablet_manager.cc:902] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:45.583554 11003 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:55509)
I20240501 19:57:45.583648 14347 tablet_copy_client.cc:308] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:45.583667 11003 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:55509: session id = 417972166f4d40ca9b4b6471a90be6af-dbc6ed9f4bf34b1caa462aa9d85c7a16
I20240501 19:57:45.583572 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:55509)
I20240501 19:57:45.583879 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:55509: session id = 417972166f4d40ca9b4b6471a90be6af-25ea8d7ab1834abfa34ff2357aa554b0
I20240501 19:57:45.584662 11003 tablet_copy_source_session.cc:205] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.584808 11002 tablet_copy_source_session.cc:205] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.587235 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:38117)
I20240501 19:57:45.587333 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 8e48d711c5714ee58ebbcbd094ebf3e7 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:38117: session id = ea26a64181784350bcd3540d0633b219-8e48d711c5714ee58ebbcbd094ebf3e7
I20240501 19:57:45.588112 11002 tablet_copy_source_session.cc:205] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.588878 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e48d711c5714ee58ebbcbd094ebf3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.590255 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.590770 14347 tablet_copy_client.cc:783] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.590942 14347 tablet_copy_client.cc:647] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.592092 14347 tablet_copy_client.cc:523] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.593420 14347 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:45.598264 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.599449 11376 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.601905 10983 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:45.602366 11242 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:45.603049 13941 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:45.603695 13941 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:45.604035 11311 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a170766f497f491ba9b45f92f2a0e751. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:45.605273 11312 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a170766f497f491ba9b45f92f2a0e751. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.606335 14339 tablet_copy_client.cc:783] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.606588 14339 tablet_copy_client.cc:647] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.606729 13941 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.609560 10982 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.610065 10837 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.618372 11242 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.618821 11245 consensus_queue.cc:237] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.620019 10983 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.620101 11112 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.620417 14253 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.620695 14253 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:45.620975 11179 consensus_peers.cc:485] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ddd522180bd4e53906cd9572f82ab67. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.624944 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.625058 14339 tablet_copy_client.cc:523] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.627074 14339 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:45.628580 14380 ts_tablet_manager.cc:902] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:45.631799 14380 tablet_copy_client.cc:308] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:45.632687 14340 tablet_copy_client.cc:783] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.636209 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.632887 14340 tablet_copy_client.cc:647] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.644381 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:45.645112 14339 log.cc:824] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:45.645380 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:45.645954 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 95b8c1a34cc94d29986289939b2159bb from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47003)
I20240501 19:57:45.646080 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 95b8c1a34cc94d29986289939b2159bb from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47003: session id = 14002a0a4c674b438c78ca5db73637cc-95b8c1a34cc94d29986289939b2159bb
I20240501 19:57:45.647357 11396 tablet_copy_source_session.cc:205] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.649139 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:45.650147 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:45.650800 11375 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.654048 11242 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.654384 14333 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.656860 11113 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.657337 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19d8d7d255494a49920849d10b800fa7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.661080 14333 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.661707 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.664175 14380 tablet_copy_client.cc:783] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.664362 14380 tablet_copy_client.cc:647] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.666638 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.669325 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.669888 14340 tablet_copy_client.cc:523] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.671118 14340 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:45.672940 14380 tablet_copy_client.cc:523] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.674480 14380 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:45.678987 14416 ts_tablet_manager.cc:902] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:45.680131 14416 tablet_copy_client.cc:308] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:45.680330 14407 ts_tablet_manager.cc:902] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:45.681378 14407 tablet_copy_client.cc:308] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:45.681785 14253 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.682946 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet c49b9c5ebaab42e18fe7bbe8127d569e from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:53481)
I20240501 19:57:45.683032 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet c49b9c5ebaab42e18fe7bbe8127d569e from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:53481: session id = ea26a64181784350bcd3540d0633b219-c49b9c5ebaab42e18fe7bbe8127d569e
I20240501 19:57:45.683884 11134 tablet_copy_source_session.cc:205] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.685441 10837 catalog_manager.cc:5484] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.687162 10983 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.691391 11112 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:45.692775 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49b9c5ebaab42e18fe7bbe8127d569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.694978 14407 tablet_copy_client.cc:783] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.695163 14407 tablet_copy_client.cc:647] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.696494 14407 tablet_copy_client.cc:523] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.697711 14407 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:45.704777 14436 ts_tablet_manager.cc:902] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:45.705354 14436 tablet_copy_client.cc:308] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:45.710687 12832 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.711407 11242 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.713207 11112 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.718353 10837 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.721733 10983 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.723922 11112 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:45.723997 11242 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:57:45.724284 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fcd6776aebf47c99f05850de83b36e6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.724613 13952 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:45.725438 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fcd6776aebf47c99f05850de83b36e6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.725466 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 2def1cda07c349c69b3abe25804d3d1d from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:45.725548 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 2def1cda07c349c69b3abe25804d3d1d from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-2def1cda07c349c69b3abe25804d3d1d
I20240501 19:57:45.725797 13952 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:45.726472 11395 tablet_copy_source_session.cc:205] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.726773 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:45.726869 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-156d2ba7ff2c403db2f63cbae4acfec1
I20240501 19:57:45.727278 13952 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.727687 11396 tablet_copy_source_session.cc:205] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.730278 10839 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.731139 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.735711 11112 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.737344 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.738162 11242 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.739976 14416 tablet_copy_client.cc:783] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.740154 14416 tablet_copy_client.cc:647] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.741636 14416 tablet_copy_client.cc:523] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.743184 14416 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
W20240501 19:57:45.745903 14291 log.cc:924] Time spent T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20240501 19:57:45.759554 14436 tablet_copy_client.cc:783] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.759850 14436 tablet_copy_client.cc:647] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.761157 14436 tablet_copy_client.cc:523] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.762179 14436 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:45.762631 14347 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.762943 14347 tablet_bootstrap.cc:492] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:45.763242 14347 ts_tablet_manager.cc:1367] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.170s user 0.020s sys 0.003s
I20240501 19:57:45.765581 14436 log.cc:824] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:45.768091 11112 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.769954 11242 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.769995 14327 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.770244 14327 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:45.770417 14327 ts_tablet_manager.cc:1367] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.204s user 0.034s sys 0.001s
I20240501 19:57:45.771893 14347 raft_consensus.cc:348] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.772120 14347 raft_consensus.cc:729] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.772400 14327 raft_consensus.cc:348] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.772482 14327 raft_consensus.cc:729] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.772994 10983 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.772981 14327 consensus_queue.cc:260] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.773844 14327 ts_tablet_manager.cc:1398] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:45.774143 14493 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.775173 14494 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.777626 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-269047cc3a504d7488747166efce71fc received from {username='slave'} at 127.30.1.71:53481
I20240501 19:57:45.777719 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session ea26a64181784350bcd3540d0633b219-269047cc3a504d7488747166efce71fc on tablet 269047cc3a504d7488747166efce71fc with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:45.779359 14347 consensus_queue.cc:260] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.779716 14347 ts_tablet_manager.cc:1398] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240501 19:57:45.780545 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-8e48d711c5714ee58ebbcbd094ebf3e7 received from {username='slave'} at 127.30.1.71:38117
I20240501 19:57:45.780616 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session ea26a64181784350bcd3540d0633b219-8e48d711c5714ee58ebbcbd094ebf3e7 on tablet 8e48d711c5714ee58ebbcbd094ebf3e7 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:45.788341 14493 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.790189 11242 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
W20240501 19:57:45.791337 11049 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f79cd7d4ec4b4697a2637e6d14a892. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.792711 10983 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.794654 10841 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:45.811079 14291 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.814134 14340 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.814527 14340 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:45.814865 14340 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.144s user 0.022s sys 0.007s
I20240501 19:57:45.814920 14380 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.815284 14380 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:45.815627 14380 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.141s user 0.020s sys 0.005s
I20240501 19:57:45.817930 14380 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.818234 14380 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.817351 14340 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.818467 14340 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.818909 14380 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.818923 14340 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.819787 14340 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:57:45.819913 14380 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:57:45.821435 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-95b8c1a34cc94d29986289939b2159bb received from {username='slave'} at 127.30.1.70:47003
I20240501 19:57:45.821527 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-95b8c1a34cc94d29986289939b2159bb on tablet 95b8c1a34cc94d29986289939b2159bb with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:45.822268 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-25ea8d7ab1834abfa34ff2357aa554b0 received from {username='slave'} at 127.30.1.69:55509
I20240501 19:57:45.822337 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-25ea8d7ab1834abfa34ff2357aa554b0 on tablet 25ea8d7ab1834abfa34ff2357aa554b0 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:45.823813 14291 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:45.823980 14291 ts_tablet_manager.cc:1367] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.258s user 0.034s sys 0.003s
I20240501 19:57:45.824487 14291 raft_consensus.cc:348] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.824570 14291 raft_consensus.cc:729] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.824712 14291 consensus_queue.cc:260] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:45.825028 14291 ts_tablet_manager.cc:1398] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:45.825840 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-0a1bbe74dbc74fa687b0a64d3aa203f3 received from {username='slave'} at 127.30.1.70:50311
I20240501 19:57:45.825907 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-0a1bbe74dbc74fa687b0a64d3aa203f3 on tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:45.834668 14339 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.834933 14339 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:45.835091 14339 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.208s user 0.034s sys 0.000s
I20240501 19:57:45.838763 14407 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.839061 14407 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:45.839228 14407 ts_tablet_manager.cc:1367] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.142s user 0.023s sys 0.002s
I20240501 19:57:45.839581 14407 raft_consensus.cc:348] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.844933 14407 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.845196 14407 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.846053 14407 ts_tablet_manager.cc:1398] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:45.846505 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-c49b9c5ebaab42e18fe7bbe8127d569e received from {username='slave'} at 127.30.1.71:53481
I20240501 19:57:45.846571 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session ea26a64181784350bcd3540d0633b219-c49b9c5ebaab42e18fe7bbe8127d569e on tablet c49b9c5ebaab42e18fe7bbe8127d569e with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:45.849241 14436 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.850945 14416 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:45.851325 14416 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:45.851671 14416 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.109s user 0.034s sys 0.000s
I20240501 19:57:45.854159 14416 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.854437 14416 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.854926 14416 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.855865 14416 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:57:45.856858 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-156d2ba7ff2c403db2f63cbae4acfec1 received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:45.856938 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-156d2ba7ff2c403db2f63cbae4acfec1 on tablet 156d2ba7ff2c403db2f63cbae4acfec1 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:45.857367 14436 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:45.857585 14436 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.095s user 0.026s sys 0.001s
I20240501 19:57:45.858055 14436 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.858665 14436 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.858853 14436 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:45.859138 14436 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:45.859607 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-2def1cda07c349c69b3abe25804d3d1d received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:45.859674 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-2def1cda07c349c69b3abe25804d3d1d on tablet 2def1cda07c349c69b3abe25804d3d1d with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:45.865295 11375 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.866778 11245 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.867000 11311 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee298406a4824f1aa2fa742b7dec1026. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.867143 13941 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.867204 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.867612 13941 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.874577 14339 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.874715 14339 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:45.874879 14339 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.875191 14339 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240501 19:57:45.875638 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-dbc6ed9f4bf34b1caa462aa9d85c7a16 received from {username='slave'} at 127.30.1.69:55509
I20240501 19:57:45.875710 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-dbc6ed9f4bf34b1caa462aa9d85c7a16 on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:45.876157 13941 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.879310 14291 ts_tablet_manager.cc:902] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:45.879348 11245 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.879580 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.879716 14291 tablet_copy_client.cc:308] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 19:57:45.881332 10837 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.882076 11375 consensus_queue.cc:237] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:45.884133 11245 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.884297 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d032fb77ac944629a45b808cc76c2f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.884553 12832 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.884706 11112 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.885231 12832 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.886548 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 318eb8fd9972435a8e51f63bac077f09 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47007)
I20240501 19:57:45.886675 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 318eb8fd9972435a8e51f63bac077f09 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47007: session id = 14002a0a4c674b438c78ca5db73637cc-318eb8fd9972435a8e51f63bac077f09
I20240501 19:57:45.888420 11265 tablet_copy_source_session.cc:205] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.890143 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.892201 14291 tablet_copy_client.cc:783] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.893046 14291 tablet_copy_client.cc:647] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.894454 12832 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.895146 14291 tablet_copy_client.cc:523] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.895172 11245 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.896289 14291 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:45.896732 11112 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.897187 11375 consensus_queue.cc:237] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:45.898478 10983 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:45.899739 10837 catalog_manager.cc:5484] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:45.900138 12832 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.901010 11245 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:45.901190 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fc8c5aab87341e3b79a8e121e6db01e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.902709 12832 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:45.913028 14436 ts_tablet_manager.cc:902] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:45.913439 14436 tablet_copy_client.cc:308] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:45.913843 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:45.913920 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:57:45.914350 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.914726 11396 tablet_copy_source_session.cc:205] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.915432 12832 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.915581 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.918509 14436 tablet_copy_client.cc:783] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.918655 14436 tablet_copy_client.cc:647] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.918610 11245 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.919476 10836 catalog_manager.cc:5484] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.920598 10983 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:45.931411 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.935449 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.936111 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:45.937134 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:45.937325 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:45.941834 12640 raft_consensus.cc:1203] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:45.944672 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:45.945194 14436 tablet_copy_client.cc:523] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:45.946460 14436 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:45.959832 12640 raft_consensus.cc:1203] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:45.982650 14416 ts_tablet_manager.cc:902] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:45.982981 14380 ts_tablet_manager.cc:902] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:45.983139 14416 tablet_copy_client.cc:308] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:45.983464 14380 tablet_copy_client.cc:308] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:45.984740 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet c3ee13ec9bb1468f90cfc184455f9fff from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47003)
I20240501 19:57:45.984841 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet c3ee13ec9bb1468f90cfc184455f9fff from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47003: session id = 14002a0a4c674b438c78ca5db73637cc-c3ee13ec9bb1468f90cfc184455f9fff
I20240501 19:57:45.985780 11396 tablet_copy_source_session.cc:205] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.986578 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee13ec9bb1468f90cfc184455f9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.988771 14380 tablet_copy_client.cc:783] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.988947 14380 tablet_copy_client.cc:647] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:45.991007 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 6fd958b083bb430a99130a18f91acdf7 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:42445)
I20240501 19:57:45.991115 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 6fd958b083bb430a99130a18f91acdf7 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:42445: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:45.992146 11002 tablet_copy_source_session.cc:205] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:45.992983 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:45.995029 14416 tablet_copy_client.cc:783] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:45.995167 14416 tablet_copy_client.cc:647] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.001080 14416 tablet_copy_client.cc:523] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.001480 14291 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.001737 14291 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:46.001943 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.002195 14416 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:46.001909 14291 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.106s user 0.022s sys 0.003s
I20240501 19:57:46.004465 14291 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.004566 14291 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.005420 14291 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.005749 14291 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:46.006567 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-318eb8fd9972435a8e51f63bac077f09 received from {username='slave'} at 127.30.1.70:47007
I20240501 19:57:46.006659 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-318eb8fd9972435a8e51f63bac077f09 on tablet 318eb8fd9972435a8e51f63bac077f09 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:46.008030 14436 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.008275 14436 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.008433 14436 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.062s user 0.024s sys 0.000s
I20240501 19:57:46.008991 14436 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.009101 14436 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.009656 14436 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.009835 11113 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.011606 10983 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.011965 14494 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.013423 11375 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.013849 14494 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.015349 14380 tablet_copy_client.cc:523] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 19:57:46.015887 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26b12620ad9a404d9ab70e8506ad1a39. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.016144 14436 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:46.016698 14380 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:46.017869 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19bfb955b9c94fdb8e0381e5b1855248 received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:46.017947 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19bfb955b9c94fdb8e0381e5b1855248 on tablet 19bfb955b9c94fdb8e0381e5b1855248 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:46.028546 11113 consensus_queue.cc:237] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.030315 11376 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.030346 11245 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.030735 14494 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:46.030941 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 219742bd987e4a1990233fa8c972c232. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.031287 14494 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.033771 11374 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.035269 11242 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.035265 11113 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.035625 14333 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:46.035761 11310 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bc8fbbdc94e49d2b8b639083e083df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.035910 12535 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.042973 14291 ts_tablet_manager.cc:902] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:46.043408 14291 tablet_copy_client.cc:308] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 19:57:46.043831 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 8ddd522180bd4e53906cd9572f82ab67 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47007)
I20240501 19:57:46.043923 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 8ddd522180bd4e53906cd9572f82ab67 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47007: session id = 14002a0a4c674b438c78ca5db73637cc-8ddd522180bd4e53906cd9572f82ab67
I20240501 19:57:46.044862 11265 tablet_copy_source_session.cc:205] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.045915 12773 raft_consensus.cc:1203] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.046723 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd522180bd4e53906cd9572f82ab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.046725 14380 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.046980 14380 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:46.047133 14380 ts_tablet_manager.cc:1367] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.003s
I20240501 19:57:46.047508 14380 raft_consensus.cc:348] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.047628 14380 raft_consensus.cc:729] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.047745 14380 consensus_queue.cc:260] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.048027 14380 ts_tablet_manager.cc:1398] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:46.048599 14291 tablet_copy_client.cc:783] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.048722 14291 tablet_copy_client.cc:647] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.049458 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-c3ee13ec9bb1468f90cfc184455f9fff received from {username='slave'} at 127.30.1.70:47003
I20240501 19:57:46.049536 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-c3ee13ec9bb1468f90cfc184455f9fff on tablet c3ee13ec9bb1468f90cfc184455f9fff with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:46.051425 14291 tablet_copy_client.cc:523] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.052518 14291 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:46.054567 13037 raft_consensus.cc:1203] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.055042 14494 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.055351 14416 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.055580 14416 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.055718 14416 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.008s
I20240501 19:57:46.057468 11375 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.058447 10834 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.059753 10983 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.063326 14407 ts_tablet_manager.cc:902] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.063755 14407 tablet_copy_client.cc:308] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.063767 14416 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.063900 14416 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.064045 14416 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.064385 14416 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:46.064750 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-6fd958b083bb430a99130a18f91acdf7 received from {username='slave'} at 127.30.1.72:42445
I20240501 19:57:46.064818 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-6fd958b083bb430a99130a18f91acdf7 on tablet 6fd958b083bb430a99130a18f91acdf7 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:46.068753 12535 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.069562 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet c3ee13ec9bb1468f90cfc184455f9fff from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:56931)
I20240501 19:57:46.069656 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet c3ee13ec9bb1468f90cfc184455f9fff from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:56931: session id = ea26a64181784350bcd3540d0633b219-c3ee13ec9bb1468f90cfc184455f9fff
I20240501 19:57:46.070452 11396 tablet_copy_source_session.cc:205] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.070852 11245 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.071751 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee13ec9bb1468f90cfc184455f9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.072867 11113 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.074553 14493 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.077212 10841 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.079350 12773 raft_consensus.cc:1203] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.080680 11245 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.081931 11376 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.085815 10836 catalog_manager.cc:5484] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.087721 14407 tablet_copy_client.cc:783] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.087852 14407 tablet_copy_client.cc:647] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.093852 12905 raft_consensus.cc:1203] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.094365 14291 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.094592 14291 tablet_bootstrap.cc:492] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:46.094744 14291 ts_tablet_manager.cc:1367] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.042s user 0.026s sys 0.000s
I20240501 19:57:46.095139 14291 raft_consensus.cc:348] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.095239 14291 raft_consensus.cc:729] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.095431 14291 consensus_queue.cc:260] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.096202 14291 ts_tablet_manager.cc:1398] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:57:46.096580 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-8ddd522180bd4e53906cd9572f82ab67 received from {username='slave'} at 127.30.1.70:47007
I20240501 19:57:46.096657 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-8ddd522180bd4e53906cd9572f82ab67 on tablet 8ddd522180bd4e53906cd9572f82ab67 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:46.099557 14407 tablet_copy_client.cc:523] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.100757 14407 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.102725 14347 ts_tablet_manager.cc:902] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.103212 14347 tablet_copy_client.cc:308] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.103690 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 19d8d7d255494a49920849d10b800fa7 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:56931)
I20240501 19:57:46.103767 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 19d8d7d255494a49920849d10b800fa7 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:56931: session id = ea26a64181784350bcd3540d0633b219-19d8d7d255494a49920849d10b800fa7
I20240501 19:57:46.104636 11396 tablet_copy_source_session.cc:205] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.105439 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.107334 14347 tablet_copy_client.cc:783] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.107439 14347 tablet_copy_client.cc:647] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.108935 12905 raft_consensus.cc:1203] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.115459 14416 ts_tablet_manager.cc:902] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.115825 14416 tablet_copy_client.cc:308] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.116228 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet a170766f497f491ba9b45f92f2a0e751 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:46.116307 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet a170766f497f491ba9b45f92f2a0e751 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-a170766f497f491ba9b45f92f2a0e751
I20240501 19:57:46.117038 11396 tablet_copy_source_session.cc:205] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.117789 14416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a170766f497f491ba9b45f92f2a0e751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.119701 14416 tablet_copy_client.cc:783] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.119834 14416 tablet_copy_client.cc:647] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.126066 14416 tablet_copy_client.cc:523] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.127085 14407 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.127167 14416 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:46.127354 14407 tablet_bootstrap.cc:492] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.128053 14407 ts_tablet_manager.cc:1367] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.003s
I20240501 19:57:46.128418 14407 raft_consensus.cc:348] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.128597 14407 raft_consensus.cc:729] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.128758 14407 consensus_queue.cc:260] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.129521 14407 ts_tablet_manager.cc:1398] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.129902 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-c3ee13ec9bb1468f90cfc184455f9fff received from {username='slave'} at 127.30.1.71:56931
I20240501 19:57:46.129974 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session ea26a64181784350bcd3540d0633b219-c3ee13ec9bb1468f90cfc184455f9fff on tablet c3ee13ec9bb1468f90cfc184455f9fff with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.145082 14347 tablet_copy_client.cc:523] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.146143 14347 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.170456 14436 ts_tablet_manager.cc:902] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:46.170935 14436 tablet_copy_client.cc:308] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:46.171497 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 9fcd6776aebf47c99f05850de83b36e6 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:42445)
I20240501 19:57:46.171600 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 9fcd6776aebf47c99f05850de83b36e6 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:42445: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-9fcd6776aebf47c99f05850de83b36e6
I20240501 19:57:46.171712 14339 ts_tablet_manager.cc:902] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.172112 14339 tablet_copy_client.cc:308] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.172580 11002 tablet_copy_source_session.cc:205] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.173115 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.173386 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.173506 14340 ts_tablet_manager.cc:902] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:46.173955 14340 tablet_copy_client.cc:308] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:46.175365 14436 tablet_copy_client.cc:783] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.175978 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 9fcd6776aebf47c99f05850de83b36e6 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:55509)
I20240501 19:57:46.176049 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 9fcd6776aebf47c99f05850de83b36e6 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:55509: session id = 417972166f4d40ca9b4b6471a90be6af-9fcd6776aebf47c99f05850de83b36e6
I20240501 19:57:46.176627 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet a170766f497f491ba9b45f92f2a0e751 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42539)
I20240501 19:57:46.176710 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet a170766f497f491ba9b45f92f2a0e751 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42539: session id = 417972166f4d40ca9b4b6471a90be6af-a170766f497f491ba9b45f92f2a0e751
I20240501 19:57:46.176846 11002 tablet_copy_source_session.cc:205] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.175498 14436 tablet_copy_client.cc:647] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.177505 11396 tablet_copy_source_session.cc:205] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.177656 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.178193 14339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a170766f497f491ba9b45f92f2a0e751. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.179627 14340 tablet_copy_client.cc:783] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.179749 14340 tablet_copy_client.cc:647] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.179970 14339 tablet_copy_client.cc:783] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.180073 14339 tablet_copy_client.cc:647] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.185542 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.186882 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.186942 14339 tablet_copy_client.cc:523] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.188098 14339 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.196240 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:46.200966 14436 tablet_copy_client.cc:523] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.202169 14436 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:46.207289 12905 raft_consensus.cc:1203] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.209542 14347 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.209842 14347 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.210008 14347 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.064s user 0.023s sys 0.003s
I20240501 19:57:46.210386 14347 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.210510 14347 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.210662 14347 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.210976 14347 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.211396 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-19d8d7d255494a49920849d10b800fa7 received from {username='slave'} at 127.30.1.71:56931
I20240501 19:57:46.211469 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session ea26a64181784350bcd3540d0633b219-19d8d7d255494a49920849d10b800fa7 on tablet 19d8d7d255494a49920849d10b800fa7 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.214458 14340 tablet_copy_client.cc:523] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.215611 14340 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.217164 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.218576 14339 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.218873 14339 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:46.219046 14339 ts_tablet_manager.cc:1367] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.004s
I20240501 19:57:46.219451 14339 raft_consensus.cc:348] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.219617 14339 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.219805 14339 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.220573 14339 ts_tablet_manager.cc:1398] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.221216 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-a170766f497f491ba9b45f92f2a0e751 received from {username='slave'} at 127.30.1.69:42539
I20240501 19:57:46.221282 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-a170766f497f491ba9b45f92f2a0e751 on tablet a170766f497f491ba9b45f92f2a0e751 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.234112 14416 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.234119 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.241262 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:46.245164 14416 tablet_bootstrap.cc:492] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.245379 14416 ts_tablet_manager.cc:1367] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.118s user 0.022s sys 0.000s
I20240501 19:57:46.246184 14416 raft_consensus.cc:348] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.246377 14416 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.246548 14416 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.247298 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:46.248374 14416 ts_tablet_manager.cc:1398] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:46.251834 11396 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-a170766f497f491ba9b45f92f2a0e751 received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:46.251897 11396 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-a170766f497f491ba9b45f92f2a0e751 on tablet a170766f497f491ba9b45f92f2a0e751 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:46.254875 11376 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.256002 10983 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.256095 11113 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.256705 12832 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.256850 14033 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:46.257445 11310 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edb640b8cc04d20b0815b1beb45d9c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.274506 11376 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.275803 11245 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.276206 11113 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.276542 13941 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.276997 13941 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.277498 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95b8c1a34cc94d29986289939b2159bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.278010 12773 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.278625 12832 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.284499 12832 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.285192 14033 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.287006 11245 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.287496 10983 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.289789 12773 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.290409 10837 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.290409 10841 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.292964 11112 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.294308 13037 raft_consensus.cc:1203] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.297441 11113 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.300987 14347 ts_tablet_manager.cc:902] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.301383 14347 tablet_copy_client.cc:308] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.301710 14407 ts_tablet_manager.cc:902] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.302192 14407 tablet_copy_client.cc:308] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.303296 11396 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet ee298406a4824f1aa2fa742b7dec1026 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:56931)
I20240501 19:57:46.303375 11396 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet ee298406a4824f1aa2fa742b7dec1026 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:56931: session id = ea26a64181784350bcd3540d0633b219-ee298406a4824f1aa2fa742b7dec1026
I20240501 19:57:46.304206 11396 tablet_copy_source_session.cc:205] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.304479 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 8fc8c5aab87341e3b79a8e121e6db01e from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:56931)
I20240501 19:57:46.304545 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 8fc8c5aab87341e3b79a8e121e6db01e from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:56931: session id = ea26a64181784350bcd3540d0633b219-8fc8c5aab87341e3b79a8e121e6db01e
I20240501 19:57:46.305346 11395 tablet_copy_source_session.cc:205] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.307211 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.309315 14347 tablet_copy_client.cc:783] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.309446 14347 tablet_copy_client.cc:647] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.311899 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8c5aab87341e3b79a8e121e6db01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.313735 14407 tablet_copy_client.cc:783] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.313861 14407 tablet_copy_client.cc:647] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.327046 14407 tablet_copy_client.cc:523] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.328143 14407 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.330729 14436 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.331040 14436 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.331676 14436 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.130s user 0.021s sys 0.004s
I20240501 19:57:46.332235 14436 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.332386 14347 tablet_copy_client.cc:523] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.332412 14436 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.332566 14436 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.333287 14436 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:46.333490 14347 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.333705 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-9fcd6776aebf47c99f05850de83b36e6 received from {username='slave'} at 127.30.1.72:42445
I20240501 19:57:46.333768 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-9fcd6776aebf47c99f05850de83b36e6 on tablet 9fcd6776aebf47c99f05850de83b36e6 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:46.339059 14340 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.339350 14340 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:46.339507 14340 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.124s user 0.022s sys 0.004s
I20240501 19:57:46.339920 14340 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.340035 14340 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.340174 14340 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.340883 14340 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.341637 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-9fcd6776aebf47c99f05850de83b36e6 received from {username='slave'} at 127.30.1.69:55509
I20240501 19:57:46.341706 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-9fcd6776aebf47c99f05850de83b36e6 on tablet 9fcd6776aebf47c99f05850de83b36e6 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.365799 12773 raft_consensus.cc:1203] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.366876 14436 ts_tablet_manager.cc:902] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:46.367200 14436 tablet_copy_client.cc:308] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:46.372524 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:56035)
I20240501 19:57:46.372612 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:56035: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-19f79cd7d4ec4b4697a2637e6d14a892
I20240501 19:57:46.373512 11134 tablet_copy_source_session.cc:205] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.375118 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.377275 14436 tablet_copy_client.cc:783] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.377465 14436 tablet_copy_client.cc:647] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.383220 14347 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.383471 14347 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.383675 14347 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.050s user 0.020s sys 0.006s
I20240501 19:57:46.384104 14347 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.384227 14347 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.384383 14347 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.384661 14347 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:46.385093 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-ee298406a4824f1aa2fa742b7dec1026 received from {username='slave'} at 127.30.1.71:56931
I20240501 19:57:46.385167 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session ea26a64181784350bcd3540d0633b219-ee298406a4824f1aa2fa742b7dec1026 on tablet ee298406a4824f1aa2fa742b7dec1026 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.388391 14407 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.388738 14407 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.389350 14407 ts_tablet_manager.cc:1367] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.005s
I20240501 19:57:46.389801 14407 raft_consensus.cc:348] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.389959 14407 raft_consensus.cc:729] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.390261 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.390458 14407 consensus_queue.cc:260] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.390836 14407 ts_tablet_manager.cc:1398] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.391654 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-8fc8c5aab87341e3b79a8e121e6db01e received from {username='slave'} at 127.30.1.71:56931
I20240501 19:57:46.391721 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session ea26a64181784350bcd3540d0633b219-8fc8c5aab87341e3b79a8e121e6db01e on tablet 8fc8c5aab87341e3b79a8e121e6db01e with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.393052 13037 raft_consensus.cc:1203] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.395939 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.398434 11376 consensus_queue.cc:237] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.399133 14436 tablet_copy_client.cc:523] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.400373 14436 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:46.401085 11245 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.401966 14333 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.402339 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d032fb77ac944629a45b808cc76c2f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.405201 11113 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.405443 11376 consensus_queue.cc:237] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.407558 13941 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.407665 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d032fb77ac944629a45b808cc76c2f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.408071 12904 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.408406 10983 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.408821 12535 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.408937 11242 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:57:46.409303 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fc8c5aab87341e3b79a8e121e6db01e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.409674 12535 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.411060 12535 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.412395 14333 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.412710 12535 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.414229 11242 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.414916 10983 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.416208 11113 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.416486 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.416967 10837 catalog_manager.cc:5484] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.418920 11242 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.421464 10836 catalog_manager.cc:5484] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.423365 13037 raft_consensus.cc:1203] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.428530 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.431674 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:46.435395 14340 ts_tablet_manager.cc:902] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:46.435748 14340 tablet_copy_client.cc:308] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:46.438156 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:46.444895 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.462435 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.462824 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.469312 12773 raft_consensus.cc:1203] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.472281 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:46.476653 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:44905)
I20240501 19:57:46.476743 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:44905: session id = 417972166f4d40ca9b4b6471a90be6af-26b12620ad9a404d9ab70e8506ad1a39
I20240501 19:57:46.477705 11134 tablet_copy_source_session.cc:205] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.478572 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.480088 12905 raft_consensus.cc:1203] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.480681 14340 tablet_copy_client.cc:783] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.480805 14340 tablet_copy_client.cc:647] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.482038 14340 tablet_copy_client.cc:523] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.483374 14340 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.483999 11376 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.485498 12905 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.485706 11242 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.485849 14333 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.486033 12832 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.487613 11113 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.487964 12832 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.488521 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19d8d7d255494a49920849d10b800fa7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.488889 14333 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.489576 12905 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.489933 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:46.491458 10837 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.492796 11242 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.495334 14407 ts_tablet_manager.cc:902] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:46.495829 11112 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.496824 14407 tablet_copy_client.cc:308] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:46.497620 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 219742bd987e4a1990233fa8c972c232 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:53481)
I20240501 19:57:46.497699 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 219742bd987e4a1990233fa8c972c232 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:53481: session id = ea26a64181784350bcd3540d0633b219-219742bd987e4a1990233fa8c972c232
I20240501 19:57:46.498565 11134 tablet_copy_source_session.cc:205] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.499320 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:46.499392 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219742bd987e4a1990233fa8c972c232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.501405 14407 tablet_copy_client.cc:783] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.501343 12904 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.501533 14407 tablet_copy_client.cc:647] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.504009 12773 raft_consensus.cc:1203] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.515139 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.515784 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.517802 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.524499 14407 tablet_copy_client.cc:523] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.525610 14407 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.533921 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:46.554360 14340 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.554688 14340 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:46.554965 14340 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.072s user 0.021s sys 0.003s
I20240501 19:57:46.555642 14340 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.555934 14340 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.556124 14340 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.556567 14340 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:46.557041 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-26b12620ad9a404d9ab70e8506ad1a39 received from {username='slave'} at 127.30.1.69:44905
I20240501 19:57:46.557121 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-26b12620ad9a404d9ab70e8506ad1a39 on tablet 26b12620ad9a404d9ab70e8506ad1a39 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.568528 11376 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.570895 11112 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.571038 10983 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.571267 14333 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.571504 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edb640b8cc04d20b0815b1beb45d9c8. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:46.572508 11310 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edb640b8cc04d20b0815b1beb45d9c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.572607 14333 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:46.573716 14436 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.574324 14436 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.574486 14436 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.174s user 0.015s sys 0.010s
I20240501 19:57:46.574913 14333 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.575325 14436 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.575420 14436 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.575528 14436 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.575783 14436 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.576761 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19f79cd7d4ec4b4697a2637e6d14a892 received from {username='slave'} at 127.30.1.72:56035
I20240501 19:57:46.576824 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19f79cd7d4ec4b4697a2637e6d14a892 on tablet 19f79cd7d4ec4b4697a2637e6d14a892 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:46.577361 11112 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.577543 10834 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.580789 13037 raft_consensus.cc:1203] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.581097 14407 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.581365 14407 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.581539 14407 ts_tablet_manager.cc:1367] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.056s user 0.025s sys 0.000s
I20240501 19:57:46.581936 14407 raft_consensus.cc:348] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.582083 14407 raft_consensus.cc:729] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.582278 14407 consensus_queue.cc:260] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.582549 14407 ts_tablet_manager.cc:1398] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:46.583959 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-219742bd987e4a1990233fa8c972c232 received from {username='slave'} at 127.30.1.71:53481
I20240501 19:57:46.584029 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session ea26a64181784350bcd3540d0633b219-219742bd987e4a1990233fa8c972c232 on tablet 219742bd987e4a1990233fa8c972c232 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.586973 10983 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:46.600828 10983 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.602422 11242 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:57:46.602483 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5faf889870246b78b82e19c041dc119. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.602792 11112 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.603142 14970 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.603286 13801 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.606065 13801 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.607744 11112 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.607861 11242 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.610323 10836 catalog_manager.cc:5484] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.613101 12640 raft_consensus.cc:1203] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.625614 14993 ts_tablet_manager.cc:902] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.626194 14993 tablet_copy_client.cc:308] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.627229 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47003)
I20240501 19:57:46.627319 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47003: session id = 14002a0a4c674b438c78ca5db73637cc-2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:46.628163 11395 tablet_copy_source_session.cc:205] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.630199 14993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.632432 14993 tablet_copy_client.cc:783] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.632710 14993 tablet_copy_client.cc:647] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.635531 14993 tablet_copy_client.cc:523] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.636505 14993 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:46.682443 14340 ts_tablet_manager.cc:902] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.682884 14340 tablet_copy_client.cc:308] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.683323 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 95b8c1a34cc94d29986289939b2159bb from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42539)
I20240501 19:57:46.683416 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 95b8c1a34cc94d29986289939b2159bb from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42539: session id = 417972166f4d40ca9b4b6471a90be6af-95b8c1a34cc94d29986289939b2159bb
I20240501 19:57:46.684286 11395 tablet_copy_source_session.cc:205] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.685482 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.686961 13037 raft_consensus.cc:1203] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.687598 14340 tablet_copy_client.cc:783] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.687739 14340 tablet_copy_client.cc:647] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.689474 14340 tablet_copy_client.cc:523] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.690646 14340 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.695330 14993 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.695588 14993 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:46.695741 14993 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.059s user 0.027s sys 0.000s
I20240501 19:57:46.696142 14993 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.696260 14993 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.696389 14993 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.697885 10983 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.699535 11376 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.699549 11242 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.699939 13801 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:46.700470 10918 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d6621e1ff294dc9a474e076873cc72c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.701311 13801 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.711638 14993 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:57:46.712065 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-2bc8fbbdc94e49d2b8b639083e083df3 received from {username='slave'} at 127.30.1.70:47003
I20240501 19:57:46.712191 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-2bc8fbbdc94e49d2b8b639083e083df3 on tablet 2bc8fbbdc94e49d2b8b639083e083df3 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:46.715377 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.719288 13801 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.721364 11242 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.721665 10841 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.723215 11376 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:46.738153 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:46.743235 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:46.745754 14340 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.746157 14340 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:46.746351 14340 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.056s user 0.026s sys 0.002s
I20240501 19:57:46.750303 14340 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.751085 14340 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.751305 14340 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.752126 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.753677 14340 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:46.759693 12640 raft_consensus.cc:1203] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:46.762516 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-95b8c1a34cc94d29986289939b2159bb received from {username='slave'} at 127.30.1.69:42539
I20240501 19:57:46.762593 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-95b8c1a34cc94d29986289939b2159bb on tablet 95b8c1a34cc94d29986289939b2159bb with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.764070 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.765951 10983 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.767910 11242 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.768723 11112 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:46.769233 13801 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.769532 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5faf889870246b78b82e19c041dc119. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.769690 13801 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:57:46.774358 10917 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5faf889870246b78b82e19c041dc119. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.774315 13801 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.776569 11242 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.778735 11112 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.778970 10834 catalog_manager.cc:5484] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 4 to 5, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.793954 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.801870 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:46.806805 14407 ts_tablet_manager.cc:902] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.807188 14407 tablet_copy_client.cc:308] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.807579 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 5d032fb77ac944629a45b808cc76c2f1 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:56931)
I20240501 19:57:46.807658 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 5d032fb77ac944629a45b808cc76c2f1 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:56931: session id = ea26a64181784350bcd3540d0633b219-5d032fb77ac944629a45b808cc76c2f1
I20240501 19:57:46.808446 11395 tablet_copy_source_session.cc:205] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.809571 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d032fb77ac944629a45b808cc76c2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.811589 14407 tablet_copy_client.cc:783] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.811740 14407 tablet_copy_client.cc:647] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.817857 14407 tablet_copy_client.cc:523] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.818862 14407 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:46.819911 14340 ts_tablet_manager.cc:902] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.820297 14340 tablet_copy_client.cc:308] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.820674 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 8fc8c5aab87341e3b79a8e121e6db01e from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42539)
I20240501 19:57:46.820755 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 8fc8c5aab87341e3b79a8e121e6db01e from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42539: session id = 417972166f4d40ca9b4b6471a90be6af-8fc8c5aab87341e3b79a8e121e6db01e
I20240501 19:57:46.821790 11395 tablet_copy_source_session.cc:205] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.823626 14340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc8c5aab87341e3b79a8e121e6db01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.839705 14340 tablet_copy_client.cc:783] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.839865 14340 tablet_copy_client.cc:647] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.839866 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:46.843075 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:46.844264 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:46.848855 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:46.860842 15138 ts_tablet_manager.cc:902] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.861347 15138 tablet_copy_client.cc:308] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.862578 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.863016 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 5d032fb77ac944629a45b808cc76c2f1 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42539)
I20240501 19:57:46.863121 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 5d032fb77ac944629a45b808cc76c2f1 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42539: session id = 417972166f4d40ca9b4b6471a90be6af-5d032fb77ac944629a45b808cc76c2f1
I20240501 19:57:46.864023 11395 tablet_copy_source_session.cc:205] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.867882 14407 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.868132 14407 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:46.868276 14407 ts_tablet_manager.cc:1367] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.000s
I20240501 19:57:46.868752 14407 raft_consensus.cc:348] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.868891 14407 raft_consensus.cc:729] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.869072 14407 consensus_queue.cc:260] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:46.869387 14407 ts_tablet_manager.cc:1398] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 19:57:46.869777 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-5d032fb77ac944629a45b808cc76c2f1 received from {username='slave'} at 127.30.1.71:56931
I20240501 19:57:46.869836 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session ea26a64181784350bcd3540d0633b219-5d032fb77ac944629a45b808cc76c2f1 on tablet 5d032fb77ac944629a45b808cc76c2f1 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.872574 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.874287 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.875766 11112 consensus_queue.cc:237] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:46.877955 11376 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.878244 11242 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.878333 14670 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.878643 14670 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.882252 12904 raft_consensus.cc:1203] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 19:57:46.889817 11048 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4f0699b5bdf44499c87ff1541cd2ad3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.893385 14340 tablet_copy_client.cc:523] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.894744 14340 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.896143 14436 ts_tablet_manager.cc:902] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.896492 14436 tablet_copy_client.cc:308] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.896939 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 19d8d7d255494a49920849d10b800fa7 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:46.897054 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 19d8d7d255494a49920849d10b800fa7 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-19d8d7d255494a49920849d10b800fa7
I20240501 19:57:46.897848 11395 tablet_copy_source_session.cc:205] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.898684 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.900800 14436 tablet_copy_client.cc:783] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.901000 14436 tablet_copy_client.cc:647] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.902966 13037 raft_consensus.cc:1203] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.906450 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.907703 15138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d032fb77ac944629a45b808cc76c2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.909914 15138 tablet_copy_client.cc:783] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.910097 15138 tablet_copy_client.cc:647] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:46.916730 12640 raft_consensus.cc:1203] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.920500 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:46.923105 15138 tablet_copy_client.cc:523] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.924255 15138 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:46.925266 14340 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.925503 14340 tablet_bootstrap.cc:492] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:46.925632 14340 ts_tablet_manager.cc:1367] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.001s
I20240501 19:57:46.926126 14340 raft_consensus.cc:348] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.926250 14340 raft_consensus.cc:729] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:46.926378 14340 consensus_queue.cc:260] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:46.926653 14340 ts_tablet_manager.cc:1398] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:46.927141 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-8fc8c5aab87341e3b79a8e121e6db01e received from {username='slave'} at 127.30.1.69:42539
I20240501 19:57:46.927211 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-8fc8c5aab87341e3b79a8e121e6db01e on tablet 8fc8c5aab87341e3b79a8e121e6db01e with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.935647 14973 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.938359 11376 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.941044 10836 catalog_manager.cc:5484] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.941920 14436 tablet_copy_client.cc:523] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.943199 14436 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:46.945438 11245 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:46.951298 12904 raft_consensus.cc:1203] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:46.966418 11245 consensus_queue.cc:237] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.968087 11112 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.968510 14265 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.969197 11376 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:46.970301 14265 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:46.976478 14993 ts_tablet_manager.cc:902] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:46.976842 14993 tablet_copy_client.cc:308] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:46.977437 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47003)
I20240501 19:57:46.977514 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47003: session id = 14002a0a4c674b438c78ca5db73637cc-6edb640b8cc04d20b0815b1beb45d9c8
I20240501 19:57:46.978371 11395 tablet_copy_source_session.cc:205] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:46.979252 14993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:46.981233 14993 tablet_copy_client.cc:783] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:46.981364 14993 tablet_copy_client.cc:647] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
W20240501 19:57:46.981822 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54d2db0f85ac4a90a1df779f89398463. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:46.982901 11376 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.984452 14265 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.986641 10836 catalog_manager.cc:5484] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.989218 11112 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:46.993631 14993 tablet_copy_client.cc:523] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:46.994798 14993 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:46.994946 14436 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:46.995185 14436 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:46.995337 14436 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.001s
I20240501 19:57:46.995836 14436 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:46.995965 14436 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.005472 14436 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.005810 14436 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 19:57:47.006631 15138 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.006973 15138 tablet_bootstrap.cc:492] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:47.007185 15138 ts_tablet_manager.cc:1367] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.083s user 0.026s sys 0.000s
I20240501 19:57:47.007840 15138 raft_consensus.cc:348] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:47.008004 15138 raft_consensus.cc:729] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.012033 12773 raft_consensus.cc:1203] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:47.017469 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19d8d7d255494a49920849d10b800fa7 received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:47.017549 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-19d8d7d255494a49920849d10b800fa7 on tablet 19d8d7d255494a49920849d10b800fa7 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:47.019032 15138 consensus_queue.cc:260] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:47.022184 15138 ts_tablet_manager.cc:1398] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 19:57:47.022864 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-5d032fb77ac944629a45b808cc76c2f1 received from {username='slave'} at 127.30.1.69:42539
I20240501 19:57:47.022938 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-5d032fb77ac944629a45b808cc76c2f1 on tablet 5d032fb77ac944629a45b808cc76c2f1 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:47.033459 14993 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.033794 14993 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:47.034000 14993 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.004s
I20240501 19:57:47.034492 14993 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:47.034669 14993 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.034849 14993 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:47.035260 14993 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:47.035780 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-6edb640b8cc04d20b0815b1beb45d9c8 received from {username='slave'} at 127.30.1.70:47003
I20240501 19:57:47.035842 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-6edb640b8cc04d20b0815b1beb45d9c8 on tablet 6edb640b8cc04d20b0815b1beb45d9c8 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:47.152878 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.162413 12640 raft_consensus.cc:1203] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.163635 14993 ts_tablet_manager.cc:902] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:47.164043 14993 tablet_copy_client.cc:308] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:47.164462 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet d5faf889870246b78b82e19c041dc119 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:50311)
I20240501 19:57:47.164564 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet d5faf889870246b78b82e19c041dc119 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:50311: session id = 14002a0a4c674b438c78ca5db73637cc-d5faf889870246b78b82e19c041dc119
I20240501 19:57:47.165544 11002 tablet_copy_source_session.cc:205] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.166332 14993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5faf889870246b78b82e19c041dc119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.166843 15138 ts_tablet_manager.cc:902] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:47.167240 15138 tablet_copy_client.cc:308] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:47.167663 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42539)
I20240501 19:57:47.167771 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42539: session id = 417972166f4d40ca9b4b6471a90be6af-6edb640b8cc04d20b0815b1beb45d9c8
I20240501 19:57:47.168557 14993 tablet_copy_client.cc:783] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.168614 11395 tablet_copy_source_session.cc:205] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.168704 14993 tablet_copy_client.cc:647] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.169485 15138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.170145 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.171675 15138 tablet_copy_client.cc:783] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.171818 15138 tablet_copy_client.cc:647] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.173560 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.187721 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.187958 14407 ts_tablet_manager.cc:902] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:47.188396 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.188395 14407 tablet_copy_client.cc:308] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:47.189009 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet d5faf889870246b78b82e19c041dc119 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:38117)
I20240501 19:57:47.189100 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet d5faf889870246b78b82e19c041dc119 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:38117: session id = ea26a64181784350bcd3540d0633b219-d5faf889870246b78b82e19c041dc119
I20240501 19:57:47.189616 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.189946 11002 tablet_copy_source_session.cc:205] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.190688 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.191018 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5faf889870246b78b82e19c041dc119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.193219 14407 tablet_copy_client.cc:783] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.193364 14407 tablet_copy_client.cc:647] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.193399 15390 ts_tablet_manager.cc:902] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:47.193843 15390 tablet_copy_client.cc:308] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:47.197209 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 2d6621e1ff294dc9a474e076873cc72c from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:38117)
I20240501 19:57:47.197284 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 2d6621e1ff294dc9a474e076873cc72c from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:38117: session id = ea26a64181784350bcd3540d0633b219-2d6621e1ff294dc9a474e076873cc72c
I20240501 19:57:47.198002 11002 tablet_copy_source_session.cc:205] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.198678 15390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6621e1ff294dc9a474e076873cc72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.200362 15390 tablet_copy_client.cc:783] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.200492 15390 tablet_copy_client.cc:647] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.202685 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.208680 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.211139 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.212577 14993 tablet_copy_client.cc:523] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.213896 14993 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:47.215032 15138 tablet_copy_client.cc:523] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.216178 15138 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:47.219790 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.221495 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.221694 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.235901 11376 consensus_queue.cc:237] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.237965 11245 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:47.238580 13941 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:57:47.240185 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eb1f0dcd4cf49cd969eb28f1eb5d9ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:47.240476 14407 tablet_copy_client.cc:523] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.241677 10983 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:57:47.243295 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.248081 13941 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:57:47.254987 15390 tablet_copy_client.cc:523] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.258121 14407 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:47.258530 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.263991 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.269917 15390 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:47.281440 14993 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.281690 14993 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:47.281836 14993 ts_tablet_manager.cc:1367] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.000s
I20240501 19:57:47.282212 14993 raft_consensus.cc:348] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.282320 14993 raft_consensus.cc:729] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.282467 14993 consensus_queue.cc:260] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.282713 14993 ts_tablet_manager.cc:1398] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:47.283099 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-d5faf889870246b78b82e19c041dc119 received from {username='slave'} at 127.30.1.70:50311
I20240501 19:57:47.283161 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-d5faf889870246b78b82e19c041dc119 on tablet d5faf889870246b78b82e19c041dc119 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:47.290246 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.301263 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.305011 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.313704 12904 raft_consensus.cc:1203] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.317811 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.322116 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.342357 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.346125 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.364655 12640 raft_consensus.cc:1203] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.367110 13037 raft_consensus.cc:1203] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.367448 15138 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.367712 15138 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:47.367869 15138 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.152s user 0.024s sys 0.000s
I20240501 19:57:47.367987 14993 ts_tablet_manager.cc:902] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:47.368402 14993 tablet_copy_client.cc:308] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:47.375627 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.368304 15138 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:47.375859 15138 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.376009 15138 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:47.376327 15138 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:57:47.379459 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-6edb640b8cc04d20b0815b1beb45d9c8 received from {username='slave'} at 127.30.1.69:42539
I20240501 19:57:47.379540 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-6edb640b8cc04d20b0815b1beb45d9c8 on tablet 6edb640b8cc04d20b0815b1beb45d9c8 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:47.381992 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet b4f0699b5bdf44499c87ff1541cd2ad3 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:55243)
I20240501 19:57:47.382091 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet b4f0699b5bdf44499c87ff1541cd2ad3 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:55243: session id = 14002a0a4c674b438c78ca5db73637cc-b4f0699b5bdf44499c87ff1541cd2ad3
I20240501 19:57:47.382948 11134 tablet_copy_source_session.cc:205] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.384528 14407 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.384804 14407 tablet_bootstrap.cc:492] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:47.385113 14407 ts_tablet_manager.cc:1367] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.127s user 0.023s sys 0.004s
I20240501 19:57:47.385237 12832 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:47.385687 14407 raft_consensus.cc:348] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.385792 14407 raft_consensus.cc:729] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.385917 14407 consensus_queue.cc:260] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.387243 14407 ts_tablet_manager.cc:1398] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:47.388139 10839 catalog_manager.cc:5484] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:47.389225 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-d5faf889870246b78b82e19c041dc119 received from {username='slave'} at 127.30.1.71:38117
I20240501 19:57:47.389302 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session ea26a64181784350bcd3540d0633b219-d5faf889870246b78b82e19c041dc119 on tablet d5faf889870246b78b82e19c041dc119 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:47.389397 10983 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:47.391044 11242 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:47.394682 14993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f0699b5bdf44499c87ff1541cd2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.396713 14993 tablet_copy_client.cc:783] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.396853 14993 tablet_copy_client.cc:647] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.398869 14993 tablet_copy_client.cc:523] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.399930 14993 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:47.417471 15390 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.417814 15390 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:47.418038 15390 ts_tablet_manager.cc:1367] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.148s user 0.026s sys 0.000s
I20240501 19:57:47.418473 14436 ts_tablet_manager.cc:902] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:47.418757 15390 raft_consensus.cc:348] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:47.418877 15390 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.418889 14436 tablet_copy_client.cc:308] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 19:57:47.419028 15390 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:47.419839 15390 ts_tablet_manager.cc:1398] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:47.420320 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-2d6621e1ff294dc9a474e076873cc72c received from {username='slave'} at 127.30.1.71:38117
I20240501 19:57:47.420387 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session ea26a64181784350bcd3540d0633b219-2d6621e1ff294dc9a474e076873cc72c on tablet 2d6621e1ff294dc9a474e076873cc72c with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:47.421404 12640 raft_consensus.cc:1203] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.432348 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 54d2db0f85ac4a90a1df779f89398463 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:32885)
I20240501 19:57:47.432433 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 54d2db0f85ac4a90a1df779f89398463 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:32885: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-54d2db0f85ac4a90a1df779f89398463
I20240501 19:57:47.434144 11265 tablet_copy_source_session.cc:205] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.435125 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54d2db0f85ac4a90a1df779f89398463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.437120 14436 tablet_copy_client.cc:783] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.437245 14436 tablet_copy_client.cc:647] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.444820 14436 tablet_copy_client.cc:523] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.445930 14436 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:47.456451 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.465704 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.467579 14993 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.467679 12773 raft_consensus.cc:1203] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.467854 14993 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:47.468014 14993 ts_tablet_manager.cc:1367] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.068s user 0.023s sys 0.000s
I20240501 19:57:47.468462 14993 raft_consensus.cc:348] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.468585 14993 raft_consensus.cc:729] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.468706 14993 consensus_queue.cc:260] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:47.468971 14993 ts_tablet_manager.cc:1398] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:47.469357 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-b4f0699b5bdf44499c87ff1541cd2ad3 received from {username='slave'} at 127.30.1.70:55243
I20240501 19:57:47.469434 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-b4f0699b5bdf44499c87ff1541cd2ad3 on tablet b4f0699b5bdf44499c87ff1541cd2ad3 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:47.488065 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.500201 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.502893 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.505498 14436 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.505793 14436 tablet_bootstrap.cc:492] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:47.505961 14436 ts_tablet_manager.cc:1367] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.060s user 0.024s sys 0.003s
I20240501 19:57:47.506595 14436 raft_consensus.cc:348] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.506705 14436 raft_consensus.cc:729] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.506815 14436 consensus_queue.cc:260] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.507110 14436 ts_tablet_manager.cc:1398] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:47.510957 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-54d2db0f85ac4a90a1df779f89398463 received from {username='slave'} at 127.30.1.72:32885
I20240501 19:57:47.511035 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-54d2db0f85ac4a90a1df779f89398463 on tablet 54d2db0f85ac4a90a1df779f89398463 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:47.515143 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.516862 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.541044 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.566881 12904 raft_consensus.cc:1203] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.621143 12904 raft_consensus.cc:1203] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:47.645176 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.646824 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.651196 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.654950 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.660166 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.662842 12773 raft_consensus.cc:1203] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.666831 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.669879 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.685443 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.698618 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.703282 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.707792 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.711114 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.720045 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.724431 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.734617 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.739158 14436 ts_tablet_manager.cc:902] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:47.739527 14436 tablet_copy_client.cc:308] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:47.740000 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:38755)
I20240501 19:57:47.740085 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:38755: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-5eb1f0dcd4cf49cd969eb28f1eb5d9ea
I20240501 19:57:47.740978 11395 tablet_copy_source_session.cc:205] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:47.741817 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:47.743822 14436 tablet_copy_client.cc:783] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:47.743952 14436 tablet_copy_client.cc:647] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:47.745126 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.746474 14436 tablet_copy_client.cc:523] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:47.747597 14436 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:47.781697 12640 raft_consensus.cc:1203] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:57:47.801584 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.803261 12773 raft_consensus.cc:1203] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:47.818256 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.818460 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.827267 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.837455 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.846225 14436 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.846467 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.846491 14436 tablet_bootstrap.cc:492] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:47.855482 13037 raft_consensus.cc:1203] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:47.858983 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.875101 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.877288 14436 ts_tablet_manager.cc:1367] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.130s user 0.021s sys 0.004s
I20240501 19:57:47.877765 14436 raft_consensus.cc:348] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.877875 14436 raft_consensus.cc:729] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:47.878034 14436 consensus_queue.cc:260] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:47.878330 14436 ts_tablet_manager.cc:1398] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:47.878989 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-5eb1f0dcd4cf49cd969eb28f1eb5d9ea received from {username='slave'} at 127.30.1.72:38755
I20240501 19:57:47.879066 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-5eb1f0dcd4cf49cd969eb28f1eb5d9ea on tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:47.909497 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.910588 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:47.910898 11112 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:47.913184 11376 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:47.913384 12640 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:47.913676 15164 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:47.917095 15164 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:47.926149 15164 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:47.926419 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.929608 11376 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:47.931862 10839 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:47.938817 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:47.949080 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:47.952543 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.952601 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:47.957352 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:47.994280 12640 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:48.004655 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b12620ad9a404d9ab70e8506ad1a39 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:48.007930 15778 tablet_replica.cc:330] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:48.008294 15778 raft_consensus.cc:2229] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.008548 15778 raft_consensus.cc:2258] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.009527 15778 ts_tablet_manager.cc:1875] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.011965 15778 ts_tablet_manager.cc:1888] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.012130 15778 log.cc:1196] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/26b12620ad9a404d9ab70e8506ad1a39
I20240501 19:57:48.012755 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 26b12620ad9a404d9ab70e8506ad1a39 (table rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]) successfully deleted
I20240501 19:57:48.035164 11375 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:48.037113 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.038487 12904 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.038880 15376 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.041182 15376 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.044520 15376 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.046258 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.048974 12904 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.049217 10838 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.052922 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet ee298406a4824f1aa2fa742b7dec1026 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:48.054374 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.060801 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.067842 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.077425 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.079905 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.091320 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.093899 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.099195 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.116544 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.133519 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.133764 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.154011 15856 tablet_replica.cc:330] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:48.154384 15856 raft_consensus.cc:2229] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.155826 15856 raft_consensus.cc:2258] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.156885 15856 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.159510 15856 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.159643 15856 log.cc:1196] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee298406a4824f1aa2fa742b7dec1026
I20240501 19:57:48.159948 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.160449 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet ee298406a4824f1aa2fa742b7dec1026 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 19:57:48.160913 13037 raft_consensus.cc:1203] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:57:48.162799 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.164290 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.172315 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.184612 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.217772 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.240142 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.242672 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.245548 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.268975 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.277788 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.278663 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.278699 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.293257 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.299834 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.303908 11376 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:48.308169 11245 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.308547 12773 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.310040 15376 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.310323 15376 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.313318 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.324143 15376 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.326742 10833 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.330667 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:48.333711 11245 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.335247 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.335561 12773 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.341742 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.343122 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.347420 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.362104 15974 tablet_replica.cc:330] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:48.365274 15974 raft_consensus.cc:2229] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.365556 15974 raft_consensus.cc:2258] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.366716 15974 ts_tablet_manager.cc:1875] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.368633 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.380867 15974 ts_tablet_manager.cc:1888] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.380968 15974 log.cc:1196] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:48.382800 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 2bc8fbbdc94e49d2b8b639083e083df3 (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:48.382804 11113 consensus_queue.cc:237] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:48.384025 12904 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.384214 10983 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.384514 15747 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.384687 15753 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.387014 15747 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.389245 10983 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.389318 10833 catalog_manager.cc:5484] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.393289 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 269047cc3a504d7488747166efce71fc with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:48.395502 12904 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:48.397614 15999 tablet_replica.cc:330] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:48.397905 15999 raft_consensus.cc:2229] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.398101 15999 raft_consensus.cc:2258] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.399072 15999 ts_tablet_manager.cc:1875] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.401698 15999 ts_tablet_manager.cc:1888] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.402225 15999 log.cc:1196] T 269047cc3a504d7488747166efce71fc P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/269047cc3a504d7488747166efce71fc
I20240501 19:57:48.403671 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 269047cc3a504d7488747166efce71fc (table rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]) successfully deleted
I20240501 19:57:48.410584 11375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47134
I20240501 19:57:48.410727 11375 raft_consensus.cc:595] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:48.445133 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.455590 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.456681 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.463641 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.475368 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.476001 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.497824 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.498404 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.558863 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.562803 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.563118 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.573984 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.575095 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.576224 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.585107 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.599887 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.600183 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.600302 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.643257 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.652386 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.656070 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.658524 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.659511 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.681851 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.683055 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.689549 11355 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 19:57:48.692037 15972 raft_consensus.cc:979] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: : Instructing follower e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:48.692169 15972 raft_consensus.cc:1067] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:48.692752 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1"
dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
from {username='slave'} at 127.30.1.68:42723
I20240501 19:57:48.692955 13037 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:48.693061 13037 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:48.694211 13037 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:48.695559 13037 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:48.696408 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:48.696521 11113 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:48.696487 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:48.696596 11242 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:48.697533 11113 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:48.697535 11242 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:48.698065 12972 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 92ecfb9f8f6745efa5aa2e168bafcc4e, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:57:48.699293 11375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:48.699402 11375 raft_consensus.cc:3032] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:48.699440 11375 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:48.699573 11375 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:48.699731 11375 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:48.700628 11375 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:48.704102 13016 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.706630 16185 raft_consensus.cc:2781] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:48.706797 16185 raft_consensus.cc:686] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Becoming Leader. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Running, Role: LEADER
I20240501 19:57:48.707095 16185 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:48.713006 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.713312 11242 consensus_queue.cc:237] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:48.714532 10983 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.714581 12773 raft_consensus.cc:1261] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.714828 15205 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.715076 15205 consensus_queue.cc:1035] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.717903 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.721251 15205 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.722918 12773 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.725838 10833 catalog_manager.cc:5484] T 8ddd522180bd4e53906cd9572f82ab67 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.728003 10839 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72). New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:48.729202 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddd522180bd4e53906cd9572f82ab67 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:48.729414 15974 tablet_replica.cc:330] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:48.729544 15974 raft_consensus.cc:2229] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.729669 15974 raft_consensus.cc:2258] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.730259 15974 ts_tablet_manager.cc:1875] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.732201 15974 ts_tablet_manager.cc:1888] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.732267 15974 log.cc:1196] T 8ddd522180bd4e53906cd9572f82ab67 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ddd522180bd4e53906cd9572f82ab67
I20240501 19:57:48.734949 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 8ddd522180bd4e53906cd9572f82ab67 (table rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]) successfully deleted
I20240501 19:57:48.738914 10983 raft_consensus.cc:2932] T 8ddd522180bd4e53906cd9572f82ab67 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:48.742388 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.745330 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.747130 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:48.781580 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:48.783711 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.789911 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.802563 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:48.808499 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:48.811093 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:48.823681 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:48.830581 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.888567 11376 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:48.890013 11113 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.890030 13037 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:48.890568 15980 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.890894 15376 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:48.895238 15376 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:48.896466 11113 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:48.897163 13037 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:48.898267 10839 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:48.902160 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:48.909509 16295 tablet_replica.cc:330] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:48.909821 16295 raft_consensus.cc:2229] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.910041 16295 raft_consensus.cc:2258] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.911711 16295 ts_tablet_manager.cc:1875] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:48.915910 16295 ts_tablet_manager.cc:1888] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:48.916029 16295 log.cc:1196] T 19bfb955b9c94fdb8e0381e5b1855248 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:57:48.918174 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 19bfb955b9c94fdb8e0381e5b1855248 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
I20240501 19:57:48.977072 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.979909 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:48.992707 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:48.996392 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.005044 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.014773 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.015219 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.017238 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.018509 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.020336 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.026396 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.036059 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.047230 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.058959 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.076272 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.077147 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.170979 11376 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:49.171772 16186 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.175884 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.184558 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.191726 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.197286 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.206773 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.207202 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.207475 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.213425 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.214018 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.222211 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.225113 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.229105 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.230966 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.232980 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.256841 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.265758 11113 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:49.266237 16186 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.271615 10983 consensus_queue.cc:237] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:49.274032 12904 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.274405 16234 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.275930 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.287304 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.296176 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.285125 11245 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:49.273201 11242 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.300369 16234 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.306622 16509 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.315189 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.317384 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.325340 16234 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.328296 12904 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.329113 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.333388 10841 catalog_manager.cc:5484] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.335901 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.338201 10982 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:49.341006 12640 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.341663 15110 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.344660 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.344798 11113 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.345186 15110 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.346617 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:49.351033 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.370525 16524 tablet_replica.cc:330] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:49.370601 11245 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.370843 16524 raft_consensus.cc:2229] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.370975 16524 raft_consensus.cc:2258] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.371439 16524 ts_tablet_manager.cc:1875] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.373641 16524 ts_tablet_manager.cc:1888] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:49.373710 16524 log.cc:1196] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/8e48d711c5714ee58ebbcbd094ebf3e7
I20240501 19:57:49.375468 15110 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.377204 11113 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.377964 10841 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.378690 12640 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.380981 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ea8d7ab1834abfa34ff2357aa554b0 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:49.383250 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 8e48d711c5714ee58ebbcbd094ebf3e7 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
I20240501 19:57:49.388686 16533 tablet_replica.cc:330] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:49.388834 16533 raft_consensus.cc:2229] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.389010 16533 raft_consensus.cc:2258] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.389511 16533 ts_tablet_manager.cc:1875] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.391618 16533 ts_tablet_manager.cc:1888] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:49.391722 16533 log.cc:1196] T 25ea8d7ab1834abfa34ff2357aa554b0 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/25ea8d7ab1834abfa34ff2357aa554b0
I20240501 19:57:49.392549 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 25ea8d7ab1834abfa34ff2357aa554b0 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 19:57:49.415942 11113 consensus_queue.cc:237] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:49.420154 12904 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.420591 11376 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.421193 15164 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.421532 15164 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.424705 16586 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.427317 11376 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.430632 10841 catalog_manager.cc:5484] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.433116 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 219742bd987e4a1990233fa8c972c232 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:49.433308 16533 tablet_replica.cc:330] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:49.433444 16533 raft_consensus.cc:2229] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.433596 16533 raft_consensus.cc:2258] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.434170 16533 ts_tablet_manager.cc:1875] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.434948 12904 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:49.436070 16533 ts_tablet_manager.cc:1888] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:49.436144 16533 log.cc:1196] T 219742bd987e4a1990233fa8c972c232 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/219742bd987e4a1990233fa8c972c232
I20240501 19:57:49.436738 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 219742bd987e4a1990233fa8c972c232 (table rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]) successfully deleted
I20240501 19:57:49.442870 11242 consensus_queue.cc:237] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:49.444439 13037 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.444792 11113 raft_consensus.cc:1261] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.444909 16612 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.445384 16188 consensus_queue.cc:1035] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.447630 11112 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:49.447752 16612 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.449263 11242 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.449806 13037 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:49.450181 15164 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.450074 13036 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.450239 10839 catalog_manager.cc:5484] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.451937 15164 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.452458 16586 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:49.453752 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 54d2db0f85ac4a90a1df779f89398463 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:49.453969 16524 tablet_replica.cc:330] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:49.454109 16524 raft_consensus.cc:2229] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.454241 16524 raft_consensus.cc:2258] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.454514 11113 raft_consensus.cc:2932] T 54d2db0f85ac4a90a1df779f89398463 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.454859 16524 ts_tablet_manager.cc:1875] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.455652 11242 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.457731 13037 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.458338 10839 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.458899 16524 ts_tablet_manager.cc:1888] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:49.458969 16524 log.cc:1196] T 54d2db0f85ac4a90a1df779f89398463 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/54d2db0f85ac4a90a1df779f89398463
I20240501 19:57:49.459591 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 54d2db0f85ac4a90a1df779f89398463 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 19:57:49.461464 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f79cd7d4ec4b4697a2637e6d14a892 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:49.465314 16642 tablet_replica.cc:330] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:49.467784 16642 raft_consensus.cc:2229] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.468041 16642 raft_consensus.cc:2258] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.468654 16642 ts_tablet_manager.cc:1875] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.470979 16642 ts_tablet_manager.cc:1888] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:49.471096 16642 log.cc:1196] T 19f79cd7d4ec4b4697a2637e6d14a892 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/19f79cd7d4ec4b4697a2637e6d14a892
I20240501 19:57:49.471644 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 19f79cd7d4ec4b4697a2637e6d14a892 (table rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]) successfully deleted
I20240501 19:57:49.522328 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.527506 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.528865 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.533052 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.536526 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.546187 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.559795 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.562001 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.574206 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.578018 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.578294 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.578791 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.583851 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.586673 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.600302 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.663950 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.664198 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.673616 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.691779 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.699349 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.700574 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.700860 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.727878 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.728160 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.745548 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47300
I20240501 19:57:49.745674 11376 raft_consensus.cc:595] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:49.746273 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.751582 11355 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 19:57:49.758072 15972 raft_consensus.cc:979] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: : Instructing follower 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 19:57:49.758147 15972 raft_consensus.cc:1067] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 19:57:49.760471 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.760444 11113 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
from {username='slave'} at 127.30.1.68:41907
I20240501 19:57:49.760591 11113 raft_consensus.cc:482] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:49.760656 11113 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:49.761875 11113 raft_consensus.cc:504] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:49.762275 11113 leader_election.cc:288] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 2 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:49.762480 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:49.762600 11376 raft_consensus.cc:3032] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:49.762634 11376 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:49.762745 11376 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:49.762873 11376 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:49.763741 11376 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 2.
I20240501 19:57:49.764174 11050 leader_election.cc:302] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:49.764330 15164 raft_consensus.cc:2781] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:49.764389 15164 raft_consensus.cc:686] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:49.764516 15164 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:49.764631 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:49.765668 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bfb955b9c94fdb8e0381e5b1855248" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
I20240501 19:57:49.765826 13037 raft_consensus.cc:3037] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:49.766870 13037 raft_consensus.cc:2454] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 2.
I20240501 19:57:49.767738 10841 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:49.779173 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.798625 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.821542 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:49.821764 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.823911 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.837463 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.840142 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:49.840351 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.845002 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.845870 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.847608 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.852202 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.899196 11113 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:49.900168 12640 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:49.900617 16586 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:49.909684 16586 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.912284 12640 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.912436 10835 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 5 to 6, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:49.916308 11113 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:49.917220 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b12620ad9a404d9ab70e8506ad1a39 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:49.917569 16524 tablet_replica.cc:330] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:49.917698 16524 raft_consensus.cc:2229] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:49.917850 16524 raft_consensus.cc:2258] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:49.918342 16524 ts_tablet_manager.cc:1875] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:49.919056 12640 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:49.919450 15164 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:57:49.919673 11048 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26b12620ad9a404d9ab70e8506ad1a39. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:49.921733 16524 ts_tablet_manager.cc:1888] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:49.921803 16524 log.cc:1196] T 26b12620ad9a404d9ab70e8506ad1a39 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/26b12620ad9a404d9ab70e8506ad1a39
I20240501 19:57:49.924872 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 26b12620ad9a404d9ab70e8506ad1a39 (table rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]) successfully deleted
I20240501 19:57:49.934715 15164 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:49.935994 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 26b12620ad9a404d9ab70e8506ad1a39 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:49.936268 12640 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:49.938239 10839 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:49.952013 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.954241 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:49.955518 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:49.959254 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:49.961771 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:49.981921 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:49.985848 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.008186 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.015726 11376 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:50.015913 13037 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:50.017501 11113 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.017509 11242 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:50.017926 16509 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.018028 12773 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.018738 15972 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.019677 16509 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:50.019768 15980 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.020563 11376 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:50.021021 16509 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.022516 11242 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:50.022533 11376 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:50.023433 12640 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.024253 15972 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.024355 10841 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:50.030535 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 156d2ba7ff2c403db2f63cbae4acfec1 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:50.030994 16940 tablet_replica.cc:330] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:50.031174 16940 raft_consensus.cc:2229] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.031383 16940 raft_consensus.cc:2258] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.033908 15972 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.036372 12640 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.039072 12773 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.039247 10840 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af reported cstate change: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.041731 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b8c1a34cc94d29986289939b2159bb with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:50.042600 10983 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.044261 12773 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.044446 12905 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.044622 16951 tablet_replica.cc:330] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:50.044610 15076 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.044873 11242 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.045315 15076 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.045703 15076 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.044744 16951 raft_consensus.cc:2229] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.046486 16951 raft_consensus.cc:2258] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.046991 16951 ts_tablet_manager.cc:1875] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.047937 15076 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.048990 16951 ts_tablet_manager.cc:1888] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:50.049069 16951 log.cc:1196] T 95b8c1a34cc94d29986289939b2159bb P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/95b8c1a34cc94d29986289939b2159bb
I20240501 19:57:50.050118 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 95b8c1a34cc94d29986289939b2159bb (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:50.050689 16940 ts_tablet_manager.cc:1875] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.050808 10838 catalog_manager.cc:5484] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.052512 12905 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.053004 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet d5faf889870246b78b82e19c041dc119 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:50.053573 16940 ts_tablet_manager.cc:1888] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:50.053474 12773 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.053647 16940 log.cc:1196] T 156d2ba7ff2c403db2f63cbae4acfec1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/156d2ba7ff2c403db2f63cbae4acfec1
I20240501 19:57:50.054342 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 156d2ba7ff2c403db2f63cbae4acfec1 (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:50.054582 16940 tablet_replica.cc:330] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:50.054728 16940 raft_consensus.cc:2229] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.055006 11242 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.059211 11113 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.063033 16940 raft_consensus.cc:2258] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.063536 16940 ts_tablet_manager.cc:1875] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.069276 16940 ts_tablet_manager.cc:1888] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:50.069350 16940 log.cc:1196] T d5faf889870246b78b82e19c041dc119 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/d5faf889870246b78b82e19c041dc119
I20240501 19:57:50.069859 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet d5faf889870246b78b82e19c041dc119 (table rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]) successfully deleted
I20240501 19:57:50.118925 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.138201 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.140070 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.154212 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.155035 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.167052 13037 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:50.167485 16586 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.173297 11376 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:50.173660 16586 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.181015 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.183140 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.208323 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.290697 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.294957 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.296036 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.297976 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.299813 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.314172 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.314872 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.323686 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.325232 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.333135 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.350594 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.352221 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.360134 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.361160 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.375408 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.376051 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.378374 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.382361 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.383946 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.385115 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.390591 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.395231 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.411511 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.430231 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.436689 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.437835 12753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.437949 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.442154 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.455374 11355 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:57:50.458835 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.462769 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.463409 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.524936 17216 ts_tablet_manager.cc:902] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:50.527016 17216 tablet_copy_client.cc:308] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:50.527415 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:55243)
I20240501 19:57:50.527490 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:55243: session id = 14002a0a4c674b438c78ca5db73637cc-26b12620ad9a404d9ab70e8506ad1a39
I20240501 19:57:50.528389 11134 tablet_copy_source_session.cc:205] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:50.529310 17216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:50.531543 17216 tablet_copy_client.cc:783] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:50.531690 17216 tablet_copy_client.cc:647] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:50.533442 17216 tablet_copy_client.cc:523] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:50.534579 17216 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:50.548260 11113 consensus_queue.cc:237] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.548985 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47426
I20240501 19:57:50.549100 11376 raft_consensus.cc:595] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:50.549503 12773 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:50.549860 16602 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:50.551146 11376 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:50.554272 16586 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:50.557693 11375 consensus_queue.cc:237] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:50.559149 11242 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.559253 12640 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.559738 12905 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.560379 16602 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.560768 16926 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.561261 12773 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.561388 16926 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.561718 16926 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.565351 10841 catalog_manager.cc:5484] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.566107 11376 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.571503 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f0699b5bdf44499c87ff1541cd2ad3 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:50.571676 17259 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.574460 12640 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.576376 12905 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.584687 10839 catalog_manager.cc:5484] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.586967 11242 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:50.590315 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc8c5aab87341e3b79a8e121e6db01e with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:50.591755 17274 tablet_replica.cc:330] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:50.591922 17274 raft_consensus.cc:2229] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.592164 17274 raft_consensus.cc:2258] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.592736 17274 ts_tablet_manager.cc:1875] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.598289 17216 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:50.598548 17216 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:50.598731 17216 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.064s user 0.022s sys 0.004s
I20240501 19:57:50.599125 17216 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.599232 17216 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:50.599356 17216 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.599625 17216 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:50.599995 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-26b12620ad9a404d9ab70e8506ad1a39 received from {username='slave'} at 127.30.1.70:55243
I20240501 19:57:50.600067 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-26b12620ad9a404d9ab70e8506ad1a39 on tablet 26b12620ad9a404d9ab70e8506ad1a39 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:50.605396 17274 ts_tablet_manager.cc:1888] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:50.605475 17274 log.cc:1196] T b4f0699b5bdf44499c87ff1541cd2ad3 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/b4f0699b5bdf44499c87ff1541cd2ad3
I20240501 19:57:50.606006 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet b4f0699b5bdf44499c87ff1541cd2ad3 (table rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]) successfully deleted
I20240501 19:57:50.606036 17289 tablet_replica.cc:330] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:50.606217 17289 raft_consensus.cc:2229] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.607230 17289 raft_consensus.cc:2258] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.607889 17289 ts_tablet_manager.cc:1875] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.610580 17289 ts_tablet_manager.cc:1888] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:50.610658 17289 log.cc:1196] T 8fc8c5aab87341e3b79a8e121e6db01e P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/8fc8c5aab87341e3b79a8e121e6db01e
I20240501 19:57:50.611196 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 8fc8c5aab87341e3b79a8e121e6db01e (table rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]) successfully deleted
I20240501 19:57:50.616801 11376 consensus_queue.cc:237] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.618780 11113 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.618790 12905 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.619284 17258 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.619398 12773 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:50.620115 17258 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.620652 17258 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:50.633936 17258 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.637007 11113 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.639034 12773 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.639560 10833 catalog_manager.cc:5484] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.642547 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ee13ec9bb1468f90cfc184455f9fff with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:50.642738 17289 tablet_replica.cc:330] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:50.642848 17289 raft_consensus.cc:2229] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.642969 17289 raft_consensus.cc:2258] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.643512 17289 ts_tablet_manager.cc:1875] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.644587 15972 raft_consensus.cc:979] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: : Instructing follower 14002a0a4c674b438c78ca5db73637cc to start an election
I20240501 19:57:50.644660 15972 raft_consensus.cc:1067] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer 14002a0a4c674b438c78ca5db73637cc to start an election
I20240501 19:57:50.645174 12773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3"
dest_uuid: "14002a0a4c674b438c78ca5db73637cc"
from {username='slave'} at 127.30.1.68:59057
I20240501 19:57:50.645344 12773 raft_consensus.cc:482] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:50.645443 12773 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:50.646312 12773 raft_consensus.cc:504] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.647408 12773 leader_election.cc:288] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 2 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:50.647614 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:50.647706 11376 raft_consensus.cc:3032] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:50.647758 11376 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:50.647888 11242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc8fbbdc94e49d2b8b639083e083df3" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:50.647883 11376 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.648015 11242 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:50.648036 11376 raft_consensus.cc:3037] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:50.648991 11376 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14002a0a4c674b438c78ca5db73637cc in term 2.
I20240501 19:57:50.649138 11242 raft_consensus.cc:2454] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14002a0a4c674b438c78ca5db73637cc in term 2.
I20240501 19:57:50.649422 17289 ts_tablet_manager.cc:1888] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:50.649487 17289 log.cc:1196] T c3ee13ec9bb1468f90cfc184455f9fff P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3ee13ec9bb1468f90cfc184455f9fff
I20240501 19:57:50.649673 12707 leader_election.cc:302] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14002a0a4c674b438c78ca5db73637cc, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 19:57:50.649909 17260 raft_consensus.cc:2781] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:50.650044 17260 raft_consensus.cc:686] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Becoming Leader. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Running, Role: LEADER
I20240501 19:57:50.650266 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet c3ee13ec9bb1468f90cfc184455f9fff (table rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]) successfully deleted
I20240501 19:57:50.650389 17260 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:50.652340 11113 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:50.653244 10841 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to 14002a0a4c674b438c78ca5db73637cc (127.30.1.70). New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.653671 12904 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:50.654052 17269 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:50.655258 11242 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:50.655694 17269 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:50.659932 17268 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:50.660396 11242 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:50.661299 12905 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:50.662237 12904 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:50.662520 10841 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:50.665431 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet c49b9c5ebaab42e18fe7bbe8127d569e with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:50.667605 17289 tablet_replica.cc:330] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:50.667759 17289 raft_consensus.cc:2229] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:50.667901 17289 raft_consensus.cc:2258] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:50.668371 17289 ts_tablet_manager.cc:1875] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:50.671458 17289 ts_tablet_manager.cc:1888] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:50.671557 17289 log.cc:1196] T c49b9c5ebaab42e18fe7bbe8127d569e P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/c49b9c5ebaab42e18fe7bbe8127d569e
I20240501 19:57:50.672361 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet c49b9c5ebaab42e18fe7bbe8127d569e (table rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]) successfully deleted
I20240501 19:57:50.700445 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.706472 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.706681 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:50.708043 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.709270 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.730443 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.741624 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.760545 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.845873 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.848668 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:50.856602 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.862612 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.884711 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:50.887164 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.906164 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.928143 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.970273 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:50.971740 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:50.977959 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:50.983649 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:50.985394 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:50.994145 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.022874 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.024751 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.025372 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:51.040131 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.043398 12772 raft_consensus.cc:1203] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:57:51.049294 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.057227 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.058476 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.059140 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.060812 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.061789 11242 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:51.062263 17348 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.074162 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.102005 11376 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:51.102586 17348 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.133546 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57286
I20240501 19:57:51.133677 11376 raft_consensus.cc:595] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:51.153012 10983 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.155037 13037 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.155359 11113 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.155997 12640 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.156342 16963 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.156723 16963 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.159929 16963 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.163023 16963 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.163621 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.166115 13037 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.169520 10836 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.176354 12640 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.178071 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.179725 13016 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.183108 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.183331 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:51.183916 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.197912 11113 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.210865 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.213562 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.241691 11376 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:51.241792 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcd6776aebf47c99f05850de83b36e6 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:51.243621 13037 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.243839 12640 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.243916 10983 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.244275 17258 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.244505 16926 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.244661 17258 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.247181 17647 tablet_replica.cc:330] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:51.247321 17647 raft_consensus.cc:2229] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:51.247467 17647 raft_consensus.cc:2258] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:51.247969 17647 ts_tablet_manager.cc:1875] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:51.250531 17647 ts_tablet_manager.cc:1888] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:51.250595 17647 log.cc:1196] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/9fcd6776aebf47c99f05850de83b36e6
I20240501 19:57:51.251487 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 9fcd6776aebf47c99f05850de83b36e6 (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:51.257720 16926 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:51.259817 13037 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:51.260128 10983 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:51.262799 12640 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:51.263741 10838 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:51.266649 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet a170766f497f491ba9b45f92f2a0e751 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:51.270402 17649 raft_consensus.cc:979] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: : Instructing follower e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:51.270476 17649 raft_consensus.cc:1067] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:51.271023 17647 tablet_replica.cc:330] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:51.271080 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d"
dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
from {username='slave'} at 127.30.1.68:42723
I20240501 19:57:51.271160 17647 raft_consensus.cc:2229] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:51.271193 13037 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:51.271239 13037 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.271308 17647 raft_consensus.cc:2258] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:51.271867 17647 ts_tablet_manager.cc:1875] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:51.271939 13037 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.272400 13037 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:51.272626 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:51.272764 11113 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.272980 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:51.273092 11376 raft_consensus.cc:3032] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:51.273142 11376 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:51.273278 11376 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.273459 11376 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.273780 11113 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.274420 11376 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.275229 12972 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:57:51.275401 16925 raft_consensus.cc:2781] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:51.275548 16925 raft_consensus.cc:686] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Becoming Leader. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Running, Role: LEADER
I20240501 19:57:51.275683 16925 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.276176 17647 ts_tablet_manager.cc:1888] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:51.276257 17647 log.cc:1196] T a170766f497f491ba9b45f92f2a0e751 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/a170766f497f491ba9b45f92f2a0e751
I20240501 19:57:51.276774 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet a170766f497f491ba9b45f92f2a0e751 (table rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]) successfully deleted
I20240501 19:57:51.278612 10840 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72). New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.281430 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:51.281549 10983 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.282496 10983 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.326409 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.327121 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.329607 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.332648 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.343073 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.347426 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.355255 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.356001 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.363649 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.363838 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.364953 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.367367 11376 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.369168 12904 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.369556 17649 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.370607 13037 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.370922 17649 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.372133 11113 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:51.372490 17649 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:51.375176 17649 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.377955 10840 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.380383 12904 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.382172 13037 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.386104 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d8d7d255494a49920849d10b800fa7 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:51.386310 17647 tablet_replica.cc:330] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:51.386449 17647 raft_consensus.cc:2229] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:51.386586 17647 raft_consensus.cc:2258] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:51.387167 17647 ts_tablet_manager.cc:1875] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:51.393065 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.393148 17647 ts_tablet_manager.cc:1888] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:51.393218 17647 log.cc:1196] T 19d8d7d255494a49920849d10b800fa7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/19d8d7d255494a49920849d10b800fa7
I20240501 19:57:51.393849 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 19d8d7d255494a49920849d10b800fa7 (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:51.394553 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.406047 11113 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.420432 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.421661 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.431460 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.480808 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.483280 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.499408 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.499647 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.509119 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.513135 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.515408 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.517731 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.519533 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.529783 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.531751 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.532258 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.548446 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.567611 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.602452 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.604221 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.631299 10983 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7"
dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45422
I20240501 19:57:51.631464 10983 raft_consensus.cc:595] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:51.669440 11376 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:51.669939 17688 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.689960 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.702512 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.705123 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.709064 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.722875 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.730299 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.739650 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.740108 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.748076 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.751049 13016 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:51.753309 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.756647 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.775132 10963 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:51.786154 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.791581 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.797624 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.821653 17604 raft_consensus.cc:979] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: : Instructing follower e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:51.821739 17604 raft_consensus.cc:1067] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Signalling peer e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:51.822273 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7"
dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
from {username='slave'} at 127.30.1.65:49101
I20240501 19:57:51.822391 13037 raft_consensus.cc:482] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:51.822438 13037 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.823424 13037 raft_consensus.cc:504] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.823836 13037 leader_election.cc:288] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:51.824116 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:51.824244 11245 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.824532 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:51.824637 10983 raft_consensus.cc:3032] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:51.824669 10983 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:51.824795 10983 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.825002 10983 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.825212 11245 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.825904 10983 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.826217 12973 leader_election.cc:302] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:57:51.826386 17688 raft_consensus.cc:2781] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:51.826522 17688 raft_consensus.cc:686] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Becoming Leader. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Running, Role: LEADER
I20240501 19:57:51.826649 17688 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:51.827819 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd958b083bb430a99130a18f91acdf7" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:51.827953 11113 raft_consensus.cc:3037] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:51.828851 11113 raft_consensus.cc:2454] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:51.830288 10834 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: term changed from 1 to 2, leader changed from 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) to e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72). New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:51.865263 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:51.867995 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:51.868388 18011 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.869179 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.875062 11113 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:51.875465 17688 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.880296 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:51.881853 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.893652 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.895969 10963 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:51.898552 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:51.927126 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:51.944430 10982 consensus_queue.cc:237] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:51.946000 11113 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:51.946789 17604 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.946918 12772 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:51.948199 17604 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.954190 10982 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:51.955576 11245 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:51.956508 17604 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.957559 12904 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:51.961220 17604 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:51.965857 17604 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:51.967718 12904 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:51.969965 10835 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:51.974362 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6621e1ff294dc9a474e076873cc72c with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:51.974764 18114 tablet_replica.cc:330] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:51.974932 18114 raft_consensus.cc:2229] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:51.975108 18114 raft_consensus.cc:2258] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:51.975694 18114 ts_tablet_manager.cc:1875] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:51.978188 18114 ts_tablet_manager.cc:1888] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:51.978297 18114 log.cc:1196] T 2d6621e1ff294dc9a474e076873cc72c P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/2d6621e1ff294dc9a474e076873cc72c
I20240501 19:57:51.978842 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 2d6621e1ff294dc9a474e076873cc72c (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:51.996829 17604 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:51.999004 11113 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.002427 10840 catalog_manager.cc:5484] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.004197 12772 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.004741 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.009831 11245 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.017750 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:52.017940 18114 tablet_replica.cc:330] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:52.018062 18114 raft_consensus.cc:2229] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.018178 18114 raft_consensus.cc:2258] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.018721 18114 ts_tablet_manager.cc:1875] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.021818 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:52.022847 18114 ts_tablet_manager.cc:1888] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:52.022923 18114 log.cc:1196] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/0a1bbe74dbc74fa687b0a64d3aa203f3
I20240501 19:57:52.025139 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
I20240501 19:57:52.031416 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.046499 10963 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.061069 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.062191 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.063766 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.064002 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.123019 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.134290 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.136806 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.148582 10963 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.159330 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.172763 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.173715 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.175809 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.177762 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:52.194820 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.202221 10982 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45580
I20240501 19:57:52.202345 10982 raft_consensus.cc:595] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:52.216147 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.216738 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:52.217445 10963 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.228765 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.228813 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.239954 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.259858 18160 raft_consensus.cc:979] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: : Instructing follower 1679f84270414f67ac67c882220e1472 to start an election
I20240501 19:57:52.259955 18160 raft_consensus.cc:1067] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Signalling peer 1679f84270414f67ac67c882220e1472 to start an election
I20240501 19:57:52.260455 11376 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "1679f84270414f67ac67c882220e1472"
from {username='slave'} at 127.30.1.65:48349
I20240501 19:57:52.260574 11376 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:52.260632 11376 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:52.261617 11376 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:52.261983 11376 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:52.262506 12640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "417972166f4d40ca9b4b6471a90be6af"
I20240501 19:57:52.262706 12640 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:52.263742 12640 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 2.
I20240501 19:57:52.263988 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:52.264091 10983 raft_consensus.cc:3032] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:52.264142 10983 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:52.264262 10983 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:52.264418 10983 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:52.265347 10983 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 2.
I20240501 19:57:52.266373 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:52.266490 11113 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:52.267405 11113 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1679f84270414f67ac67c882220e1472 in term 2.
I20240501 19:57:52.267768 11312 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 417972166f4d40ca9b4b6471a90be6af, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:52.267925 17649 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:52.267993 17649 raft_consensus.cc:686] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Becoming Leader. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:52.268116 17649 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:52.279707 10839 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 reported cstate change: term changed from 1 to 2, leader changed from 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) to 1679f84270414f67ac67c882220e1472 (127.30.1.68). New cstate: current_term: 2 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.289386 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.292001 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.300082 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.306715 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.306939 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.326623 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.337862 11245 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.338270 18046 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.342290 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.347174 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.362982 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.366444 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.373683 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.389218 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.390304 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.399039 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.399554 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.426836 10983 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.427222 18046 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.430133 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.433473 11113 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.436410 18046 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.444368 10983 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:52.445490 13037 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:52.445674 12640 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:52.445842 17604 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:52.446090 17582 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:52.449270 17582 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.450438 13037 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.452425 12640 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.453123 10840 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.453526 11376 consensus_queue.cc:237] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:52.455773 11245 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.457721 12904 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.457942 17649 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.458631 17649 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.460693 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fcd6776aebf47c99f05850de83b36e6 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:52.461011 18401 tablet_replica.cc:330] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:52.461206 18401 raft_consensus.cc:2229] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.461469 18401 raft_consensus.cc:2258] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.462069 18401 ts_tablet_manager.cc:1875] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.465025 12640 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.465698 18395 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.468309 18401 ts_tablet_manager.cc:1888] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:52.468391 18401 log.cc:1196] T 9fcd6776aebf47c99f05850de83b36e6 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/9fcd6776aebf47c99f05850de83b36e6
I20240501 19:57:52.469559 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 9fcd6776aebf47c99f05850de83b36e6 (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:52.470296 18395 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.472153 11245 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.473008 12904 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.473261 10838 catalog_manager.cc:5484] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.478081 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d032fb77ac944629a45b808cc76c2f1 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:52.478322 18401 tablet_replica.cc:330] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:52.478478 18401 raft_consensus.cc:2229] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.478612 18401 raft_consensus.cc:2258] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.479389 18401 ts_tablet_manager.cc:1875] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.481199 12640 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:52.482008 18401 ts_tablet_manager.cc:1888] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:52.482137 18401 log.cc:1196] T 5d032fb77ac944629a45b808cc76c2f1 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d032fb77ac944629a45b808cc76c2f1
I20240501 19:57:52.483328 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 5d032fb77ac944629a45b808cc76c2f1 (table rebalance_test_table_5 [id=70cc64f197e4430781d4667986e6ba7b]) successfully deleted
I20240501 19:57:52.497077 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.499238 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.502132 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.511940 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.514075 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.515636 11376 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:52.517189 12773 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.517628 18394 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.518939 11113 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.518967 12640 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:52.519433 16926 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.520399 16926 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.521286 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.533877 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.537441 16926 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.538213 12773 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.540081 11113 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.542205 10836 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.544850 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edb640b8cc04d20b0815b1beb45d9c8 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:52.546635 18465 tablet_replica.cc:330] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:52.546833 18465 raft_consensus.cc:2229] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.547017 18465 raft_consensus.cc:2258] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.547689 18465 ts_tablet_manager.cc:1875] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.550173 18465 ts_tablet_manager.cc:1888] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:52.550287 18465 log.cc:1196] T 6edb640b8cc04d20b0815b1beb45d9c8 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/6edb640b8cc04d20b0815b1beb45d9c8
I20240501 19:57:52.550843 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 6edb640b8cc04d20b0815b1beb45d9c8 (table rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]) successfully deleted
I20240501 19:57:52.555207 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.571659 12640 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.626420 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.636312 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.649536 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.664278 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.664888 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.668731 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.679538 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.679658 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.687517 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.705724 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.706889 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.717653 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.720163 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.720486 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.729127 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.740608 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.744240 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.761622 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.767467 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.778714 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:52.793170 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.800455 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.808521 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.812088 10983 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.812438 18394 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.820230 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.853538 12640 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.854346 18394 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.856536 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:57:52.856951 18394 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:52.874107 12773 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:52.875377 11245 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:52.875762 17909 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:52.879570 17909 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.879927 11245 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.883038 10839 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:52.886562 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:52.892032 18689 tablet_replica.cc:330] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:52.893085 18689 raft_consensus.cc:2229] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.893270 18689 raft_consensus.cc:2258] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.894345 18689 ts_tablet_manager.cc:1875] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.894353 13036 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:52.895975 11113 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:52.896384 18359 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.896766 18689 ts_tablet_manager.cc:1888] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:52.896934 18689 log.cc:1196] T 2bc8fbbdc94e49d2b8b639083e083df3 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:52.898025 11245 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:52.898159 18359 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.898422 18357 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:52.899205 11113 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.901120 10833 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.903586 12773 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:52.903872 11245 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:52.905071 11242 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:52.905845 18687 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:52.907572 18687 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.908289 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd958b083bb430a99130a18f91acdf7 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:52.908335 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 2bc8fbbdc94e49d2b8b639083e083df3 (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:52.908483 18465 tablet_replica.cc:330] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:52.908594 18465 raft_consensus.cc:2229] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.908708 18465 raft_consensus.cc:2258] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.909236 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2bc8fbbdc94e49d2b8b639083e083df3 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:52.909554 18465 ts_tablet_manager.cc:1875] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:52.909929 10840 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.910892 11242 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:52.911511 18465 ts_tablet_manager.cc:1888] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:52.911578 18465 log.cc:1196] T 6fd958b083bb430a99130a18f91acdf7 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:52.912173 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 6fd958b083bb430a99130a18f91acdf7 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
W20240501 19:57:52.912830 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bc8fbbdc94e49d2b8b639083e083df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:52.935761 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:52.936496 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:52.936774 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.945387 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:52.946187 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.950522 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:52.970052 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:52.977079 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.006357 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.016124 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.016757 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.022783 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.031155 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.041293 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.044279 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.058988 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.140146 13036 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:53.141690 11376 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:53.141991 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:53.142571 18046 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:53.142840 18046 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:53.149820 18357 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.152748 10839 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.153446 11376 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.156267 10983 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.156662 11376 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:53.156775 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def1cda07c349c69b3abe25804d3d1d with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:53.157982 12773 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:53.159262 12640 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:53.159534 18395 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.160645 18394 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.163018 18394 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.165511 10837 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.157178 18879 tablet_replica.cc:330] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:53.168109 18879 raft_consensus.cc:2229] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.168272 18879 raft_consensus.cc:2258] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.168807 18879 ts_tablet_manager.cc:1875] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.169028 12640 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.169256 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edb640b8cc04d20b0815b1beb45d9c8 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:53.170840 12773 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.171428 18879 ts_tablet_manager.cc:1888] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:53.171494 18879 log.cc:1196] T 2def1cda07c349c69b3abe25804d3d1d P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/2def1cda07c349c69b3abe25804d3d1d
I20240501 19:57:53.171976 18879 tablet_replica.cc:330] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:53.172009 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 2def1cda07c349c69b3abe25804d3d1d (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:53.172116 18879 raft_consensus.cc:2229] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.176188 18879 raft_consensus.cc:2258] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.177953 18879 ts_tablet_manager.cc:1875] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.179569 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.183848 18879 ts_tablet_manager.cc:1888] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:53.183914 18879 log.cc:1196] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/6edb640b8cc04d20b0815b1beb45d9c8
I20240501 19:57:53.184366 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 6edb640b8cc04d20b0815b1beb45d9c8 (table rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]) successfully deleted
I20240501 19:57:53.190472 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.197193 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.199262 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.200822 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.206540 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.219223 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.228806 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.233548 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.246126 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.249226 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.251999 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.259351 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.267141 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.286161 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.291728 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.309738 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.338016 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.339497 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.344812 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.346884 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.351383 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.372886 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.383984 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.425659 11376 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:53.427251 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:53.427600 18395 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:53.429464 12640 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:53.429704 18395 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.430493 16926 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:53.431429 12640 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.432238 11113 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.432515 10833 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:53.437157 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:53.440860 19007 tablet_replica.cc:330] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:53.441193 19007 raft_consensus.cc:2229] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.441450 19007 raft_consensus.cc:2258] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.441751 13036 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:53.442504 19007 ts_tablet_manager.cc:1875] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.442782 11245 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:53.443861 18359 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.445192 19007 ts_tablet_manager.cc:1888] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 19:57:53.446777 19007 log.cc:1196] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbc6ed9f4bf34b1caa462aa9d85c7a16
I20240501 19:57:53.447348 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:53.454432 18046 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.454883 11245 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.457389 10832 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.459990 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 156d2ba7ff2c403db2f63cbae4acfec1 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:53.462033 19039 tablet_replica.cc:330] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:53.462218 19039 raft_consensus.cc:2229] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.462468 19039 raft_consensus.cc:2258] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.463202 19039 ts_tablet_manager.cc:1875] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.466234 19039 ts_tablet_manager.cc:1888] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:53.466377 19039 log.cc:1196] T 156d2ba7ff2c403db2f63cbae4acfec1 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/156d2ba7ff2c403db2f63cbae4acfec1
I20240501 19:57:53.466938 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 156d2ba7ff2c403db2f63cbae4acfec1 (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:53.469923 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.474581 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.486163 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.493291 13036 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:53.494444 11245 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:53.494896 18046 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:57:53.496834 12973 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): Couldn't send request to peer 99348cf9a56645808d00c459fb76a6f8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 156d2ba7ff2c403db2f63cbae4acfec1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:53.497684 18046 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:53.500229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.500329 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 156d2ba7ff2c403db2f63cbae4acfec1 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:53.500309 11242 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:53.501726 11245 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:53.507958 10983 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:53.508311 18967 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:53.512228 12773 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:53.512995 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.513365 10840 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 7 to 8, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:53.513738 19079 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:53.516350 18967 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.516366 10983 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.520404 19097 ts_tablet_manager.cc:902] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 19:57:53.520956 19097 tablet_copy_client.cc:308] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 19:57:53.521561 10840 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.523906 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 318eb8fd9972435a8e51f63bac077f09 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:53.524085 19039 tablet_replica.cc:330] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:53.524190 19039 raft_consensus.cc:2229] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.524310 19039 raft_consensus.cc:2258] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.524794 19039 ts_tablet_manager.cc:1875] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.526650 19039 ts_tablet_manager.cc:1888] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:53.526721 19039 log.cc:1196] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/318eb8fd9972435a8e51f63bac077f09
I20240501 19:57:53.527205 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 318eb8fd9972435a8e51f63bac077f09 (table rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]) successfully deleted
I20240501 19:57:53.540386 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:58191)
I20240501 19:57:53.540504 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:58191: session id = 417972166f4d40ca9b4b6471a90be6af-2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:53.541872 12793 tablet_copy_source_session.cc:205] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:53.543339 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:53.545861 19097 tablet_copy_client.cc:783] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:53.550621 19097 tablet_copy_client.cc:647] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:53.550621 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.553485 12773 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.557379 19097 tablet_copy_client.cc:523] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:53.558764 19097 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:53.566002 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.592826 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.597337 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.609195 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.609558 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.614778 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.617115 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.624956 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.636385 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.637849 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.640323 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.647898 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.657088 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.662703 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.673527 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.686523 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.689679 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.710723 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.752329 19097 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:53.752653 19097 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:53.752858 19097 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.194s user 0.028s sys 0.001s
I20240501 19:57:53.753300 19097 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:53.753415 19097 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:53.753576 19097 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:53.754392 19097 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:53.754914 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-2bc8fbbdc94e49d2b8b639083e083df3 received from {username='slave'} at 127.30.1.69:58191
I20240501 19:57:53.754992 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-2bc8fbbdc94e49d2b8b639083e083df3 on tablet 2bc8fbbdc94e49d2b8b639083e083df3 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:53.777802 11376 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:53.779552 13036 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:53.780005 12905 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:53.780450 11113 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:53.780493 18395 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.780773 18395 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.781850 13036 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:53.782650 19176 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.783471 18395 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.784406 19176 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.786134 10833 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.786322 13036 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.786455 12905 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.786994 13037 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.789069 10834 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 5 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.795897 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:53.796113 19039 tablet_replica.cc:330] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:53.796322 19039 raft_consensus.cc:2229] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.796420 19039 raft_consensus.cc:2258] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.796885 19039 ts_tablet_manager.cc:1875] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.797644 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d8d7d255494a49920849d10b800fa7 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:53.798094 11113 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:53.799088 19039 ts_tablet_manager.cc:1888] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:53.799149 19039 log.cc:1196] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:57:53.799446 13037 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:53.799679 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 19bfb955b9c94fdb8e0381e5b1855248 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
I20240501 19:57:53.800076 17896 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:57:53.800243 11048 consensus_peers.cc:485] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19bfb955b9c94fdb8e0381e5b1855248. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:53.802367 19176 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.804517 13036 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.804772 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 19bfb955b9c94fdb8e0381e5b1855248 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:53.805583 10833 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 7 to 8, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:53.809846 19277 tablet_replica.cc:330] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:53.809979 19277 raft_consensus.cc:2229] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.810144 19277 raft_consensus.cc:2258] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.811205 13036 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:53.811787 19277 ts_tablet_manager.cc:1875] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.812183 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:53.812549 18359 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:53.815930 18359 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.817548 10982 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.817986 10835 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 6 to 7, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:53.820197 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def1cda07c349c69b3abe25804d3d1d with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:53.820394 19039 tablet_replica.cc:330] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:53.820524 19039 raft_consensus.cc:2229] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:53.820636 19039 raft_consensus.cc:2258] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:53.821365 13036 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:53.821821 19039 ts_tablet_manager.cc:1875] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:53.822932 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:53.823563 18046 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:53.823872 19039 ts_tablet_manager.cc:1888] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:53.823937 19039 log.cc:1196] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/2def1cda07c349c69b3abe25804d3d1d
W20240501 19:57:53.824669 12973 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): Couldn't send request to peer 1679f84270414f67ac67c882220e1472. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:53.826046 18046 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:53.826792 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 2def1cda07c349c69b3abe25804d3d1d (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:53.828161 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2def1cda07c349c69b3abe25804d3d1d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:53.828219 10835 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 7 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:53.828437 10982 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:53.832888 19277 ts_tablet_manager.cc:1888] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:53.832996 19277 log.cc:1196] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/19d8d7d255494a49920849d10b800fa7
I20240501 19:57:53.833649 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 19d8d7d255494a49920849d10b800fa7 (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:53.937721 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.948222 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.952657 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.953619 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:53.954989 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:53.955478 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:53.958129 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.971215 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:53.977109 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.979658 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:53.980122 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:53.990214 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.008004 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.017700 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.022740 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.038852 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.051961 19504 ts_tablet_manager.cc:902] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:54.052727 19504 tablet_copy_client.cc:308] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:54.053414 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer 99348cf9a56645808d00c459fb76a6f8 ({username='slave'} at 127.30.1.65:49101)
I20240501 19:57:54.053525 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer 99348cf9a56645808d00c459fb76a6f8 at {username='slave'} at 127.30.1.65:49101: session id = 99348cf9a56645808d00c459fb76a6f8-156d2ba7ff2c403db2f63cbae4acfec1
I20240501 19:57:54.054905 13057 tablet_copy_source_session.cc:205] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:54.060272 19504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:54.062516 19504 tablet_copy_client.cc:783] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:54.062696 19504 tablet_copy_client.cc:647] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:54.064422 19504 tablet_copy_client.cc:523] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:54.065897 19504 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:54.088047 12640 raft_consensus.cc:1203] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:54.097765 19504 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:54.098049 19504 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap complete.
I20240501 19:57:54.098201 19504 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240501 19:57:54.098577 19504 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:54.098711 19504 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:54.098814 19504 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:54.118805 19504 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 19:57:54.125042 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 99348cf9a56645808d00c459fb76a6f8-156d2ba7ff2c403db2f63cbae4acfec1 received from {username='slave'} at 127.30.1.65:49101
I20240501 19:57:54.125154 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 99348cf9a56645808d00c459fb76a6f8-156d2ba7ff2c403db2f63cbae4acfec1 on tablet 156d2ba7ff2c403db2f63cbae4acfec1 with peer 99348cf9a56645808d00c459fb76a6f8
I20240501 19:57:54.136041 10982 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:54.137226 12773 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:54.137346 12905 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:54.137576 19165 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.137854 19165 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.141800 13036 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:54.142868 11113 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:54.143277 18359 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.152037 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.156608 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.165360 18357 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.165815 11113 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.168445 10841 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.170429 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.170850 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.172176 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd958b083bb430a99130a18f91acdf7 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:54.172143 13036 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.177407 11113 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:54.178355 18357 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:54.180306 11355 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 19:57:54.185734 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.186338 19165 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.188431 12773 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.190650 18357 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.191073 11113 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.192423 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6fd958b083bb430a99130a18f91acdf7 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:54.192981 12905 raft_consensus.cc:2932] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.195150 10834 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.196403 10836 catalog_manager.cc:5484] T d5faf889870246b78b82e19c041dc119 P ea26a64181784350bcd3540d0633b219 reported cstate change: config changed from index 6 to 7, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.196826 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:54.199821 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet d5faf889870246b78b82e19c041dc119 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:54.200098 19615 tablet_replica.cc:330] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:54.202723 19615 raft_consensus.cc:2229] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:54.204862 19615 raft_consensus.cc:2258] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:54.206123 19615 ts_tablet_manager.cc:1875] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:54.211508 19615 ts_tablet_manager.cc:1888] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
W20240501 19:57:54.213356 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd958b083bb430a99130a18f91acdf7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:54.213611 19615 log.cc:1196] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:54.213976 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.214356 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 6fd958b083bb430a99130a18f91acdf7 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:54.214591 19615 tablet_replica.cc:330] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:54.214764 19615 raft_consensus.cc:2229] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:54.214919 19615 raft_consensus.cc:2258] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:54.215559 19615 ts_tablet_manager.cc:1875] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:54.217504 19615 ts_tablet_manager.cc:1888] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:54.217619 19615 log.cc:1196] T d5faf889870246b78b82e19c041dc119 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5faf889870246b78b82e19c041dc119
I20240501 19:57:54.218062 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet d5faf889870246b78b82e19c041dc119 (table rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]) successfully deleted
I20240501 19:57:54.273890 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57874
I20240501 19:57:54.274015 11376 raft_consensus.cc:595] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:57:54.280306 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.293886 18394 raft_consensus.cc:979] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: : Instructing follower 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 19:57:54.293965 18394 raft_consensus.cc:1067] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Signalling peer 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 19:57:54.294365 11113 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
from {username='slave'} at 127.30.1.68:41907
I20240501 19:57:54.294474 11113 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:54.294524 11113 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:54.295670 11113 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.295984 11113 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:54.296440 12640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "417972166f4d40ca9b4b6471a90be6af"
I20240501 19:57:54.296550 12640 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:54.297600 12640 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 3.
I20240501 19:57:54.297919 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:54.298012 11376 raft_consensus.cc:3032] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:54.298046 11376 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:54.298170 11376 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.298288 11376 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:54.299254 11376 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 3.
I20240501 19:57:54.299465 11050 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417972166f4d40ca9b4b6471a90be6af, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:54.299611 19279 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:54.299681 19279 raft_consensus.cc:686] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:54.299798 19279 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.302234 10841 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 2 to 3, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 3 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.307670 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.311921 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.318351 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.321950 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.326043 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.326826 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.333137 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.334403 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.346578 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.348001 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.348608 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.351948 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.361670 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.378525 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.382125 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.407406 19745 ts_tablet_manager.cc:902] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:54.412526 19755 ts_tablet_manager.cc:902] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:54.413589 19755 tablet_copy_client.cc:308] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:54.415347 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 2def1cda07c349c69b3abe25804d3d1d from peer 1679f84270414f67ac67c882220e1472 ({username='slave'} at 127.30.1.68:42723)
I20240501 19:57:54.415448 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 2def1cda07c349c69b3abe25804d3d1d from peer 1679f84270414f67ac67c882220e1472 at {username='slave'} at 127.30.1.68:42723: session id = 1679f84270414f67ac67c882220e1472-2def1cda07c349c69b3abe25804d3d1d
I20240501 19:57:54.416347 13057 tablet_copy_source_session.cc:205] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:54.419890 19755 ts_tablet_manager.cc:1875] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:54.421089 19745 tablet_copy_client.cc:308] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:54.421541 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:55243)
I20240501 19:57:54.421648 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:55243: session id = 14002a0a4c674b438c78ca5db73637cc-19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:57:54.422515 11134 tablet_copy_source_session.cc:205] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:54.423418 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:54.426106 19745 tablet_copy_client.cc:783] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:54.426237 19745 tablet_copy_client.cc:647] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:54.441397 19745 tablet_copy_client.cc:523] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:54.442567 19745 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:54.453202 19755 ts_tablet_manager.cc:1888] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 19:57:54.453377 19755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:54.456087 10982 raft_consensus.cc:1203] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:54.456769 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.475175 19755 tablet_copy_client.cc:783] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:54.475353 19755 tablet_copy_client.cc:647] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:54.477486 19745 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:54.477746 19745 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:54.477960 19745 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.000s
I20240501 19:57:54.478412 19745 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:54.478534 19745 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:54.478677 19745 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:54.480582 19745 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:54.481491 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-19bfb955b9c94fdb8e0381e5b1855248 received from {username='slave'} at 127.30.1.70:55243
I20240501 19:57:54.481560 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-19bfb955b9c94fdb8e0381e5b1855248 on tablet 19bfb955b9c94fdb8e0381e5b1855248 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:54.484059 19755 tablet_copy_client.cc:523] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:54.492781 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.495900 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.497391 19755 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:54.502297 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.510432 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.530398 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.538065 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.538324 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.547940 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.549587 11113 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:54.551324 12905 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:54.551792 17896 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:54.556727 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.564817 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.568343 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.577076 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.582742 17896 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:54.584894 12905 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:54.586525 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.588430 10832 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 5 to 6, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:54.590904 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet c49b9c5ebaab42e18fe7bbe8127d569e with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:54.591104 19615 tablet_replica.cc:330] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:54.591238 19615 raft_consensus.cc:2229] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:54.591363 19615 raft_consensus.cc:2258] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:54.591419 11113 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.591835 19615 ts_tablet_manager.cc:1875] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:54.592936 12905 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:54.593374 17896 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.593966 19615 ts_tablet_manager.cc:1888] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:54.594033 19615 log.cc:1196] T c49b9c5ebaab42e18fe7bbe8127d569e P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/c49b9c5ebaab42e18fe7bbe8127d569e
W20240501 19:57:54.594352 11050 consensus_peers.cc:485] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c49b9c5ebaab42e18fe7bbe8127d569e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:54.595104 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet c49b9c5ebaab42e18fe7bbe8127d569e (table rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]) successfully deleted
I20240501 19:57:54.596504 17896 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.598008 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c49b9c5ebaab42e18fe7bbe8127d569e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:54.598479 12905 raft_consensus.cc:2932] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.598793 10834 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.601718 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.609243 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.621313 11245 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:54.622681 12773 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:54.623135 19079 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:54.625993 19855 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.626348 12773 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.628443 10834 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 5 to 6, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:54.630784 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 318eb8fd9972435a8e51f63bac077f09 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:54.631965 11245 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.633628 19856 tablet_replica.cc:330] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:54.633622 12773 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:54.633823 19856 raft_consensus.cc:2229] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:54.634078 19855 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.633992 19856 raft_consensus.cc:2258] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:54.635108 19856 ts_tablet_manager.cc:1875] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:54.635443 19079 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.635816 12773 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.637411 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 318eb8fd9972435a8e51f63bac077f09 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:54.638027 10832 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 6 to 7, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:54.639654 19856 ts_tablet_manager.cc:1888] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:54.639725 19856 log.cc:1196] T 318eb8fd9972435a8e51f63bac077f09 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/318eb8fd9972435a8e51f63bac077f09
I20240501 19:57:54.640215 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 318eb8fd9972435a8e51f63bac077f09 (table rebalance_test_table_6 [id=d01e436b10394cc1bf201b8df8009d19]) successfully deleted
W20240501 19:57:54.643257 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 318eb8fd9972435a8e51f63bac077f09. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:54.658592 19755 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:54.658923 19755 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Bootstrap complete.
I20240501 19:57:54.659116 19755 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.162s user 0.029s sys 0.000s
I20240501 19:57:54.659781 19755 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:54.659888 19755 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:54.660096 19755 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:54.661347 19755 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:54.661775 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 1679f84270414f67ac67c882220e1472-2def1cda07c349c69b3abe25804d3d1d received from {username='slave'} at 127.30.1.68:42723
I20240501 19:57:54.661850 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 1679f84270414f67ac67c882220e1472-2def1cda07c349c69b3abe25804d3d1d on tablet 2def1cda07c349c69b3abe25804d3d1d with peer 1679f84270414f67ac67c882220e1472
I20240501 19:57:54.679772 11376 consensus_queue.cc:237] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:54.681286 13036 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:54.681800 11245 raft_consensus.cc:1261] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:57:54.683795 18395 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:54.685194 18395 consensus_queue.cc:1035] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:57:54.688974 18395 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.691303 13036 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.692737 10834 catalog_manager.cc:5484] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.692868 11245 raft_consensus.cc:2932] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:54.696050 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:54.696310 19856 tablet_replica.cc:330] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:54.696465 19856 raft_consensus.cc:2229] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:54.696605 19856 raft_consensus.cc:2258] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:54.697190 19856 ts_tablet_manager.cc:1875] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:54.699245 19856 ts_tablet_manager.cc:1888] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:57:54.699299 19856 log.cc:1196] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/5eb1f0dcd4cf49cd969eb28f1eb5d9ea
I20240501 19:57:54.700971 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea (table rebalance_test_table_4 [id=6be62a82bf214958a448a800bf9690b5]) successfully deleted
I20240501 19:57:54.743688 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.750852 20005 ts_tablet_manager.cc:902] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:54.751323 20005 tablet_copy_client.cc:308] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:54.761233 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.766510 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.767908 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.772964 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.786846 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.805104 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.806085 12773 raft_consensus.cc:1203] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:54.811990 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.818353 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 6fd958b083bb430a99130a18f91acdf7 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:42405)
I20240501 19:57:54.818436 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 6fd958b083bb430a99130a18f91acdf7 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:42405: session id = 417972166f4d40ca9b4b6471a90be6af-6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:54.819303 13057 tablet_copy_source_session.cc:205] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:54.819481 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.819777 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.820214 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:54.820755 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.822273 20005 tablet_copy_client.cc:783] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:54.822412 20005 tablet_copy_client.cc:647] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:54.823727 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:54.823917 20005 tablet_copy_client.cc:523] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:54.826678 20005 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:54.827025 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.844054 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.855988 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:54.869195 11376 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 19:57:54.869547 19279 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.871690 12640 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240501 19:57:54.872200 19838 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:54.884198 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.898211 20005 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:54.898684 20005 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:54.898876 20005 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.072s user 0.027s sys 0.000s
I20240501 19:57:54.899286 20005 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.899422 20005 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:54.899597 20005 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:54.899986 20005 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:54.900416 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-6fd958b083bb430a99130a18f91acdf7 received from {username='slave'} at 127.30.1.69:42405
I20240501 19:57:54.900485 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-6fd958b083bb430a99130a18f91acdf7 on tablet 6fd958b083bb430a99130a18f91acdf7 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:54.909826 11113 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44184
I20240501 19:57:54.909976 11113 raft_consensus.cc:595] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:54.943295 11376 raft_consensus.cc:1203] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:54.970746 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.977543 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:54.980563 20005 ts_tablet_manager.cc:902] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:54.980978 20005 tablet_copy_client.cc:308] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:54.981349 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet c49b9c5ebaab42e18fe7bbe8127d569e from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:44905)
I20240501 19:57:54.981438 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet c49b9c5ebaab42e18fe7bbe8127d569e from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:44905: session id = 417972166f4d40ca9b4b6471a90be6af-c49b9c5ebaab42e18fe7bbe8127d569e
I20240501 19:57:54.982488 11134 tablet_copy_source_session.cc:205] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:54.983359 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49b9c5ebaab42e18fe7bbe8127d569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:54.986085 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:54.986817 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:54.987869 20005 tablet_copy_client.cc:783] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:54.988013 20005 tablet_copy_client.cc:647] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:54.991426 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:54.997223 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.007140 20005 tablet_copy_client.cc:523] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:55.008036 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.008460 20005 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:55.008868 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.025982 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.033074 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.039414 11092 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:55.039662 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.062803 20005 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:55.063066 20005 tablet_bootstrap.cc:492] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:55.063241 20005 ts_tablet_manager.cc:1367] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.055s user 0.026s sys 0.001s
I20240501 19:57:55.063719 20005 raft_consensus.cc:348] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.063829 20005 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:55.063961 20005 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.064852 20005 ts_tablet_manager.cc:1398] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:55.065245 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-c49b9c5ebaab42e18fe7bbe8127d569e received from {username='slave'} at 127.30.1.69:44905
I20240501 19:57:55.065328 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-c49b9c5ebaab42e18fe7bbe8127d569e on tablet c49b9c5ebaab42e18fe7bbe8127d569e with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:55.065402 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.065929 13016 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.087689 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.090320 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.148075 17896 raft_consensus.cc:979] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: : Instructing follower e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:55.148176 17896 raft_consensus.cc:1067] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Signalling peer e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:55.149803 13036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892"
dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
from {username='slave'} at 127.30.1.66:47755
I20240501 19:57:55.149920 13036 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:55.149983 13036 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:55.150990 13036 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:55.151377 13036 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:55.152186 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
I20240501 19:57:55.152220 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.152298 11245 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:55.156419 11245 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:55.156831 12971 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:57:55.157339 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:55.157363 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.157446 11113 raft_consensus.cc:3032] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:55.157485 11113 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:55.157626 11113 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:55.157809 11113 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:55.158778 11113 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:55.159250 18046 raft_consensus.cc:2781] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:55.159399 18046 raft_consensus.cc:686] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Becoming Leader. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Running, Role: LEADER
I20240501 19:57:55.159543 18046 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:55.161209 10840 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: term changed from 1 to 2, leader changed from 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) to e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72). New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.167057 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.170141 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.170509 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.170682 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.172652 20005 ts_tablet_manager.cc:902] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:55.173036 20005 tablet_copy_client.cc:308] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 19:57:55.181820 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 318eb8fd9972435a8e51f63bac077f09 from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:47607)
I20240501 19:57:55.181921 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 318eb8fd9972435a8e51f63bac077f09 from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:47607: session id = 417972166f4d40ca9b4b6471a90be6af-318eb8fd9972435a8e51f63bac077f09
I20240501 19:57:55.182823 11265 tablet_copy_source_session.cc:205] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:55.184118 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:55.186178 20005 tablet_copy_client.cc:783] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:55.186316 20005 tablet_copy_client.cc:647] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:55.203694 20005 tablet_copy_client.cc:523] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:55.204926 20005 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:55.207151 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.208324 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.249264 20005 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:55.249516 20005 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:55.249696 20005 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.003s
I20240501 19:57:55.250126 20005 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.250283 20005 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:55.250435 20005 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.250865 20005 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:55.251333 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-318eb8fd9972435a8e51f63bac077f09 received from {username='slave'} at 127.30.1.69:47607
I20240501 19:57:55.251406 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-318eb8fd9972435a8e51f63bac077f09 on tablet 318eb8fd9972435a8e51f63bac077f09 with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:55.269752 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.270237 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.271272 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.276623 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.285921 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.297792 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.301151 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.314468 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.331494 11113 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.332680 12640 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 19:57:55.333070 19838 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:55.334620 19838 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.337265 10839 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 3 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.339072 12640 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.341256 11376 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.342533 12773 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:55.342969 19925 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:55.345846 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:55.346127 20333 tablet_replica.cc:330] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:55.346427 12640 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:55.346596 11113 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:55.348423 12640 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240501 19:57:55.349475 17896 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:55.349747 20333 raft_consensus.cc:2229] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:55.349942 20333 raft_consensus.cc:2258] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:55.348330 19925 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:57:55.351023 11049 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbc6ed9f4bf34b1caa462aa9d85c7a16. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:55.351189 20333 ts_tablet_manager.cc:1875] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:55.351604 19925 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.353426 20333 ts_tablet_manager.cc:1888] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240501 19:57:55.353495 20333 log.cc:1196] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/dbc6ed9f4bf34b1caa462aa9d85c7a16
I20240501 19:57:55.354008 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:55.353950 10841 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.356818 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b8c1a34cc94d29986289939b2159bb with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:55.357101 19279 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.358326 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:55.358420 12641 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.359313 10841 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 8 to 9, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 3 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.359673 12773 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.365625 20340 tablet_replica.cc:330] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:55.365805 20340 raft_consensus.cc:2229] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:55.366187 20340 raft_consensus.cc:2258] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:55.366366 12640 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.366848 20340 ts_tablet_manager.cc:1875] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:55.373555 12640 raft_consensus.cc:1203] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:55.374110 20340 ts_tablet_manager.cc:1888] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:57:55.374243 20340 log.cc:1196] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/95b8c1a34cc94d29986289939b2159bb
I20240501 19:57:55.374910 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 95b8c1a34cc94d29986289939b2159bb (table rebalance_test_table_1 [id=08fecbecd680417eaf65c3e7ba272052]) successfully deleted
I20240501 19:57:55.432718 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.455185 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.455592 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.460804 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.464547 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.476763 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.488054 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.490128 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.497113 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.505195 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.509208 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.522871 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.523334 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.524013 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.533098 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.536095 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.565763 12640 raft_consensus.cc:1203] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:57:55.570493 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.571812 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.572556 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.591846 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.597096 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.599254 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.606256 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.610265 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.612818 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.618741 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.620661 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.625141 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.627565 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.639694 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.648646 12640 raft_consensus.cc:1203] T 318eb8fd9972435a8e51f63bac077f09 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:57:55.649447 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.675735 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.727932 13036 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:55.729209 12640 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:55.729622 20240 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:55.731379 20240 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.731741 12640 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.734125 10834 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af reported cstate change: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:55.738415 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd958b083bb430a99130a18f91acdf7 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:55.738670 20340 tablet_replica.cc:330] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:55.738808 20340 raft_consensus.cc:2229] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:55.738987 20340 raft_consensus.cc:2258] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:55.739099 13036 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:55.739626 20340 ts_tablet_manager.cc:1875] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:55.740854 12640 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:57:55.741227 20240 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 19:57:55.741819 12971 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): Couldn't send request to peer 4456bc4917c644f8b69b4f5bfb8e1166. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:55.743137 20340 ts_tablet_manager.cc:1888] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 19:57:55.743290 20340 log.cc:1196] T 6fd958b083bb430a99130a18f91acdf7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:55.743856 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 6fd958b083bb430a99130a18f91acdf7 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:55.744797 11113 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:55.746062 20570 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:55.751806 20240 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:55.753324 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6fd958b083bb430a99130a18f91acdf7 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:55.754245 12640 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:55.754357 10834 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:55.759330 11245 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:55.759788 20595 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:55.770975 10982 consensus_queue.cc:237] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:55.772166 12773 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:55.772503 19626 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:55.774526 19626 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:55.774839 12773 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:55.776785 10836 catalog_manager.cc:5484] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:55.779609 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:55.779799 20340 tablet_replica.cc:330] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:55.779912 20340 raft_consensus.cc:2229] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:55.780045 20340 raft_consensus.cc:2258] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:55.780084 10982 consensus_queue.cc:237] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:55.780555 20340 ts_tablet_manager.cc:1875] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:55.782640 20340 ts_tablet_manager.cc:1888] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:55.782703 20340 log.cc:1196] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/0a1bbe74dbc74fa687b0a64d3aa203f3
I20240501 19:57:55.783197 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
W20240501 19:57:55.783315 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a1bbe74dbc74fa687b0a64d3aa203f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:55.784394 12773 raft_consensus.cc:1261] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:57:55.784737 19626 consensus_queue.cc:1035] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:55.786505 19626 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.787765 10828 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:55.788599 12773 raft_consensus.cc:2932] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.789788 10836 catalog_manager.cc:5484] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:55.852504 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.855738 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.858436 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.868414 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.872778 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.875456 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.881474 20698 ts_tablet_manager.cc:902] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:55.882028 20698 tablet_copy_client.cc:308] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:55.883515 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:56035)
I20240501 19:57:55.883610 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:56035: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-dbc6ed9f4bf34b1caa462aa9d85c7a16
I20240501 19:57:55.884466 11134 tablet_copy_source_session.cc:205] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:55.900162 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.903748 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.921299 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:55.923135 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.923521 20698 tablet_copy_client.cc:783] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:55.923668 20698 tablet_copy_client.cc:647] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:55.927372 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.929490 20698 tablet_copy_client.cc:523] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:55.930635 20698 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:55.938074 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.943827 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:55.944270 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.944764 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:55.949052 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.960747 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:55.961056 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.964514 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:55.965240 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:55.974748 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:55.984764 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:55.986163 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:56.011771 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.023733 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.046913 20698 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:56.047416 20698 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:56.047669 20698 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.117s user 0.028s sys 0.000s
I20240501 19:57:56.048126 20698 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.048242 20698 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:56.048379 20698 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.049376 20698 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:56.049811 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-dbc6ed9f4bf34b1caa462aa9d85c7a16 received from {username='slave'} at 127.30.1.72:56035
I20240501 19:57:56.049902 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-dbc6ed9f4bf34b1caa462aa9d85c7a16 on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:56.137600 11113 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:56.139093 12773 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:56.139492 13036 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:56.139621 17896 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:57:56.139874 11050 consensus_peers.cc:485] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): Couldn't send request to peer 1679f84270414f67ac67c882220e1472. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.140028 17896 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:56.143781 17896 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:56.146031 13036 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:56.148084 12773 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:56.148428 10838 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:57:56.153659 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.161180 20698 ts_tablet_manager.cc:902] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:56.161584 20698 tablet_copy_client.cc:308] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:56.161834 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.162287 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:42445)
I20240501 19:57:56.162384 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:42445: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-0a1bbe74dbc74fa687b0a64d3aa203f3
I20240501 19:57:56.163161 11002 tablet_copy_source_session.cc:205] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.163980 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1bbe74dbc74fa687b0a64d3aa203f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.166090 20698 tablet_copy_client.cc:783] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.166218 20698 tablet_copy_client.cc:647] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:56.175033 20698 tablet_copy_client.cc:523] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:56.176710 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:56.177315 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.178053 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.178390 20698 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:56.180353 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.186929 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.188524 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:56.198335 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.209424 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.217885 20698 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:56.218149 20698 tablet_bootstrap.cc:492] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:56.218338 20698 ts_tablet_manager.cc:1367] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.000s
I20240501 19:57:56.218719 20698 raft_consensus.cc:348] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.218864 20698 raft_consensus.cc:729] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:56.218987 20698 consensus_queue.cc:260] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.219308 20698 ts_tablet_manager.cc:1398] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:56.219724 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-0a1bbe74dbc74fa687b0a64d3aa203f3 received from {username='slave'} at 127.30.1.72:42445
I20240501 19:57:56.219800 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-0a1bbe74dbc74fa687b0a64d3aa203f3 on tablet 0a1bbe74dbc74fa687b0a64d3aa203f3 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:56.233441 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.252346 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:56.256825 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.273767 20923 ts_tablet_manager.cc:902] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:56.274665 20923 tablet_copy_client.cc:308] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:56.275244 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 6fd958b083bb430a99130a18f91acdf7 from peer 4456bc4917c644f8b69b4f5bfb8e1166 ({username='slave'} at 127.30.1.67:56651)
I20240501 19:57:56.275333 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 6fd958b083bb430a99130a18f91acdf7 from peer 4456bc4917c644f8b69b4f5bfb8e1166 at {username='slave'} at 127.30.1.67:56651: session id = 4456bc4917c644f8b69b4f5bfb8e1166-6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:56.275667 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:56.276283 13057 tablet_copy_source_session.cc:205] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.277271 10982 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:56.280017 12640 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.281039 20923 ts_tablet_manager.cc:1875] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:56.282330 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.283324 20923 ts_tablet_manager.cc:1888] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 19:57:56.283504 20923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd958b083bb430a99130a18f91acdf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.286614 20663 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.288846 20923 tablet_copy_client.cc:783] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.289011 20923 tablet_copy_client.cc:647] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:56.296089 20923 tablet_copy_client.cc:523] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:56.296474 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.298030 20923 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:56.305609 11225 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:57:56.324328 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.328454 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.329198 20663 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:56.330709 12640 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:56.336495 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.336599 10838 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 5 to 6, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:56.337929 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.340271 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ea8d7ab1834abfa34ff2357aa554b0 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:56.340678 10983 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:56.344566 12640 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:57:56.344857 10917 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25ea8d7ab1834abfa34ff2357aa554b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.345052 20941 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:56.346885 12640 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.348829 20941 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.351516 10828 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 25ea8d7ab1834abfa34ff2357aa554b0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:56.352535 10983 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c"
dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46268
I20240501 19:57:56.352636 10983 raft_consensus.cc:595] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:56.353130 10838 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 6 to 7, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.358749 13036 raft_consensus.cc:1203] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240501 19:57:56.360656 20946 tablet_replica.cc:330] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:56.360819 20946 raft_consensus.cc:2229] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:56.361048 20946 raft_consensus.cc:2258] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:56.361603 20946 ts_tablet_manager.cc:1875] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:56.364037 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:56.364152 20946 ts_tablet_manager.cc:1888] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:57:56.364246 20946 log.cc:1196] T 25ea8d7ab1834abfa34ff2357aa554b0 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/25ea8d7ab1834abfa34ff2357aa554b0
I20240501 19:57:56.364763 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 25ea8d7ab1834abfa34ff2357aa554b0 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 19:57:56.367216 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.371723 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.391942 20923 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:56.392272 20923 tablet_bootstrap.cc:492] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap complete.
I20240501 19:57:56.392446 20923 ts_tablet_manager.cc:1367] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.094s user 0.019s sys 0.005s
I20240501 19:57:56.392853 20923 raft_consensus.cc:348] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:56.392992 20923 raft_consensus.cc:729] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:56.393136 20923 consensus_queue.cc:260] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:56.393410 20923 ts_tablet_manager.cc:1398] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:57:56.394018 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-6fd958b083bb430a99130a18f91acdf7 received from {username='slave'} at 127.30.1.67:56651
I20240501 19:57:56.394109 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-6fd958b083bb430a99130a18f91acdf7 on tablet 6fd958b083bb430a99130a18f91acdf7 with peer 4456bc4917c644f8b69b4f5bfb8e1166
I20240501 19:57:56.407932 11113 consensus_queue.cc:237] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.409171 11376 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.409672 12773 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.410547 20861 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:57:56.410734 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4f0699b5bdf44499c87ff1541cd2ad3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.410846 20867 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.417918 20861 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.418354 12773 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.419816 11376 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.422248 10839 catalog_manager.cc:5484] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.445820 13036 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.446970 11242 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:56.447511 20596 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:56.448077 20953 raft_consensus.cc:979] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: : Instructing follower 4456bc4917c644f8b69b4f5bfb8e1166 to start an election
I20240501 19:57:56.448176 20953 raft_consensus.cc:1067] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Signalling peer 4456bc4917c644f8b69b4f5bfb8e1166 to start an election
I20240501 19:57:56.448659 11244 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c"
dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166"
from {username='slave'} at 127.30.1.65:52539
I20240501 19:57:56.448782 11244 raft_consensus.cc:482] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:56.448843 11244 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:56.449882 11244 raft_consensus.cc:504] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.450330 11244 leader_election.cc:288] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 2 election: Requested vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843)
I20240501 19:57:56.450525 20596 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.451053 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:56.451174 10983 raft_consensus.cc:3032] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:56.451213 10983 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:57:56.451324 10983 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.451462 10983 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:56.452419 10983 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 2.
I20240501 19:57:56.453125 11181 leader_election.cc:302] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:56.453294 19864 raft_consensus.cc:2781] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:56.453434 19864 raft_consensus.cc:686] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 19:57:56.453553 19864 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.453652 10841 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 5 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.453713 11244 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.456399 11092 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f79cd7d4ec4b4697a2637e6d14a892 with delete_type TABLET_DATA_TOMBSTONED (TS 92ecfb9f8f6745efa5aa2e168bafcc4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54130
I20240501 19:57:56.456661 20946 tablet_replica.cc:330] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: stopping tablet replica
I20240501 19:57:56.456826 20946 raft_consensus.cc:2229] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:56.456954 20946 raft_consensus.cc:2258] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:56.459020 20946 ts_tablet_manager.cc:1875] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:56.460773 13036 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.461884 10836 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 1 to 2, leader changed from 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 2 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.463176 11244 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:56.463593 21044 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:56.465402 20596 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.466519 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 19f79cd7d4ec4b4697a2637e6d14a892 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:56.467334 10836 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 7 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:56.469637 11244 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
W20240501 19:57:56.470480 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f79cd7d4ec4b4697a2637e6d14a892. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.470607 20946 ts_tablet_manager.cc:1888] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:56.470674 20946 log.cc:1196] T 19f79cd7d4ec4b4697a2637e6d14a892 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-1/wal/wals/19f79cd7d4ec4b4697a2637e6d14a892
I20240501 19:57:56.471205 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6621e1ff294dc9a474e076873cc72c" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ea26a64181784350bcd3540d0633b219"
I20240501 19:57:56.471792 10827 catalog_manager.cc:4830] TS 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): tablet 19f79cd7d4ec4b4697a2637e6d14a892 (table rebalance_test_table_0 [id=b6fd66afac434216ace3febe00939103]) successfully deleted
I20240501 19:57:56.471695 12905 raft_consensus.cc:3037] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:56.473757 12905 raft_consensus.cc:2454] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 2.
I20240501 19:57:56.480887 10983 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:56.482759 13036 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:56.482843 12640 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:56.483119 20953 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:56.483310 20938 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:57:56.484774 10917 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): Couldn't send request to peer 4456bc4917c644f8b69b4f5bfb8e1166. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.494338 20953 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:56.499099 10836 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 7 to 8, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:56.500628 12640 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:56.502758 13036 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 19:57:56.511549 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.523069 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.523712 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.532277 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.540812 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.545210 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.549325 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:56.553587 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.554641 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.555757 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.562436 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:56.575078 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.578716 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.581207 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.588479 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.596158 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.598315 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.609150 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.612489 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.633184 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:56.642112 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.643226 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.651927 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.655496 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.661712 13036 raft_consensus.cc:1203] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:57:56.664230 21152 ts_tablet_manager.cc:902] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:56.664775 21152 tablet_copy_client.cc:308] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:56.665252 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer 1679f84270414f67ac67c882220e1472 ({username='slave'} at 127.30.1.68:41907)
I20240501 19:57:56.665334 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 19bfb955b9c94fdb8e0381e5b1855248 from peer 1679f84270414f67ac67c882220e1472 at {username='slave'} at 127.30.1.68:41907: session id = 1679f84270414f67ac67c882220e1472-19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:57:56.666126 11134 tablet_copy_source_session.cc:205] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.666815 21152 ts_tablet_manager.cc:1875] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:56.669297 21152 ts_tablet_manager.cc:1888] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240501 19:57:56.669517 21152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bfb955b9c94fdb8e0381e5b1855248. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.671023 21152 tablet_copy_client.cc:783] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.671221 21152 tablet_copy_client.cc:647] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:56.672346 21152 tablet_copy_client.cc:523] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:56.674628 21152 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 19:57:56.701836 21152 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:56.702193 21152 tablet_bootstrap.cc:492] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Bootstrap complete.
I20240501 19:57:56.702363 21152 ts_tablet_manager.cc:1367] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.001s
I20240501 19:57:56.702720 21152 raft_consensus.cc:348] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:56.702831 21152 raft_consensus.cc:729] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 19:57:56.702957 21152 consensus_queue.cc:260] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:57:56.703208 21152 ts_tablet_manager.cc:1398] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:56.703562 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 1679f84270414f67ac67c882220e1472-19bfb955b9c94fdb8e0381e5b1855248 received from {username='slave'} at 127.30.1.68:41907
I20240501 19:57:56.703636 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 1679f84270414f67ac67c882220e1472-19bfb955b9c94fdb8e0381e5b1855248 on tablet 19bfb955b9c94fdb8e0381e5b1855248 with peer 1679f84270414f67ac67c882220e1472
I20240501 19:57:56.735967 11376 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:56.738229 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:56.739264 11310 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee298406a4824f1aa2fa742b7dec1026. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.739426 18394 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.741884 12905 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.742632 21179 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.765641 21179 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.768221 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.771225 12905 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.771509 10839 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 5 to 6, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:56.820605 11245 raft_consensus.cc:1203] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 19:57:56.840507 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.853041 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:56.854569 21290 ts_tablet_manager.cc:902] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:56.855741 21290 tablet_copy_client.cc:308] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:56.856703 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet b4f0699b5bdf44499c87ff1541cd2ad3 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:53481)
I20240501 19:57:56.856794 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet b4f0699b5bdf44499c87ff1541cd2ad3 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:53481: session id = ea26a64181784350bcd3540d0633b219-b4f0699b5bdf44499c87ff1541cd2ad3
I20240501 19:57:56.857789 11134 tablet_copy_source_session.cc:205] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.858018 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.858665 21290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f0699b5bdf44499c87ff1541cd2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.861016 21290 tablet_copy_client.cc:783] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.861202 21290 tablet_copy_client.cc:647] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:56.863564 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.867974 21290 tablet_copy_client.cc:523] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:56.869238 21290 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:56.884420 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:56.891034 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.892750 11113 consensus_queue.cc:237] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:56.893060 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.894682 12905 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.895067 20867 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:57:56.895895 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 219742bd987e4a1990233fa8c972c232. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:56.896651 11376 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:56.897044 20867 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.898113 21290 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:56.898420 21290 tablet_bootstrap.cc:492] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:56.898605 21290 ts_tablet_manager.cc:1367] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.002s
I20240501 19:57:56.899050 21290 raft_consensus.cc:348] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.899216 21290 raft_consensus.cc:729] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:56.899370 21290 consensus_queue.cc:260] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:56.901764 21290 ts_tablet_manager.cc:1398] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:57:56.903952 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-b4f0699b5bdf44499c87ff1541cd2ad3 received from {username='slave'} at 127.30.1.71:53481
I20240501 19:57:56.904017 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session ea26a64181784350bcd3540d0633b219-b4f0699b5bdf44499c87ff1541cd2ad3 on tablet b4f0699b5bdf44499c87ff1541cd2ad3 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:56.904914 21290 ts_tablet_manager.cc:902] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:56.905292 21290 tablet_copy_client.cc:308] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:56.906620 20867 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.908437 11376 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.908491 12905 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.910810 10841 catalog_manager.cc:5484] T 219742bd987e4a1990233fa8c972c232 P ea26a64181784350bcd3540d0633b219 reported cstate change: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:56.917670 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.948568 10983 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:56.949062 21056 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.964776 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:56.980854 12905 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:57:56.982722 21056 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:56.983141 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:56.985451 21338 ts_tablet_manager.cc:902] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:56.985970 21338 tablet_copy_client.cc:308] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:56.986562 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 9fcd6776aebf47c99f05850de83b36e6 from peer 4456bc4917c644f8b69b4f5bfb8e1166 ({username='slave'} at 127.30.1.67:33181)
I20240501 19:57:56.986652 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 9fcd6776aebf47c99f05850de83b36e6 from peer 4456bc4917c644f8b69b4f5bfb8e1166 at {username='slave'} at 127.30.1.67:33181: session id = 4456bc4917c644f8b69b4f5bfb8e1166-9fcd6776aebf47c99f05850de83b36e6
I20240501 19:57:56.987247 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:56.987473 11002 tablet_copy_source_session.cc:205] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.988544 21338 ts_tablet_manager.cc:1875] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:56.988550 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:56.990653 21338 ts_tablet_manager.cc:1888] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 19:57:56.990811 21338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fcd6776aebf47c99f05850de83b36e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.992614 21338 tablet_copy_client.cc:783] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.992750 21338 tablet_copy_client.cc:647] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:56.993110 21333 ts_tablet_manager.cc:902] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:57:56.993520 21333 tablet_copy_client.cc:308] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 19:57:56.993885 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:50311)
I20240501 19:57:56.993968 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:50311: session id = 14002a0a4c674b438c78ca5db73637cc-25ea8d7ab1834abfa34ff2357aa554b0
I20240501 19:57:56.994731 11002 tablet_copy_source_session.cc:205] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:56.995422 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:56.997990 21333 tablet_copy_client.cc:783] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:56.998149 21333 tablet_copy_client.cc:647] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.016613 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:55777)
I20240501 19:57:57.016729 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:55777: session id = ea26a64181784350bcd3540d0633b219-19f79cd7d4ec4b4697a2637e6d14a892
I20240501 19:57:57.017576 13057 tablet_copy_source_session.cc:205] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.018376 21290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.020282 21290 tablet_copy_client.cc:783] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.020408 21290 tablet_copy_client.cc:647] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.021526 21290 tablet_copy_client.cc:523] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.022533 21290 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:57.032438 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.034242 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.039584 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:57.053690 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.054711 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:57.060794 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.062435 21290 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.062680 21290 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:57.062835 21290 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.001s
I20240501 19:57:57.063232 21290 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:57.063364 21290 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.063489 21290 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:57.064211 21290 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:57:57.064618 21338 tablet_copy_client.cc:523] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.065233 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-19f79cd7d4ec4b4697a2637e6d14a892 received from {username='slave'} at 127.30.1.71:55777
I20240501 19:57:57.065300 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session ea26a64181784350bcd3540d0633b219-19f79cd7d4ec4b4697a2637e6d14a892 on tablet 19f79cd7d4ec4b4697a2637e6d14a892 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:57.066047 21338 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 19:57:57.068251 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.068720 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:57.085002 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.103777 21333 tablet_copy_client.cc:523] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.105448 21333 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:57.110960 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.121803 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.128232 11376 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
W20240501 19:57:57.129889 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): Couldn't send request to peer 92ecfb9f8f6745efa5aa2e168bafcc4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.130298 13037 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:57.130722 12905 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:57.131083 18394 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:57.131430 18394 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:57.132499 21338 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.137331 21338 tablet_bootstrap.cc:492] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap complete.
I20240501 19:57:57.137504 21338 ts_tablet_manager.cc:1367] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.071s user 0.026s sys 0.000s
I20240501 19:57:57.137944 21338 raft_consensus.cc:348] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:57.138079 21338 raft_consensus.cc:729] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.138197 21338 consensus_queue.cc:260] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 19:57:57.138499 21338 ts_tablet_manager.cc:1398] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:57.138931 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-9fcd6776aebf47c99f05850de83b36e6 received from {username='slave'} at 127.30.1.67:33181
I20240501 19:57:57.139025 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-9fcd6776aebf47c99f05850de83b36e6 on tablet 9fcd6776aebf47c99f05850de83b36e6 with peer 4456bc4917c644f8b69b4f5bfb8e1166
I20240501 19:57:57.144148 21229 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.147110 12905 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.148439 13037 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.150471 10841 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P ea26a64181784350bcd3540d0633b219 reported cstate change: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.155158 11376 raft_consensus.cc:1203] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 19:57:57.160305 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.175624 11113 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44586
I20240501 19:57:57.175776 11113 raft_consensus.cc:595] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Received request to transfer leadership
I20240501 19:57:57.181489 21333 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.181828 21333 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:57.182004 21333 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.077s user 0.021s sys 0.006s
I20240501 19:57:57.182411 21333 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:57.182520 21333 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.182952 21333 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:57.183260 21333 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:57.184486 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-25ea8d7ab1834abfa34ff2357aa554b0 received from {username='slave'} at 127.30.1.70:50311
I20240501 19:57:57.184579 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-25ea8d7ab1834abfa34ff2357aa554b0 on tablet 25ea8d7ab1834abfa34ff2357aa554b0 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:57.203799 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58356
I20240501 19:57:57.203940 11376 raft_consensus.cc:595] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:57.221060 11113 consensus_queue.cc:237] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.222810 12905 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:57:57.223696 10983 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:57:57.224261 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 269047cc3a504d7488747166efce71fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.224457 20861 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:57.225137 20861 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:57:57.233043 20861 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.235862 12905 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.236176 10983 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.237811 10835 catalog_manager.cc:5484] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 5 to 6, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.257879 21333 ts_tablet_manager.cc:902] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:57.258293 21333 tablet_copy_client.cc:308] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:57.258729 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet ee298406a4824f1aa2fa742b7dec1026 from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:47003)
I20240501 19:57:57.258831 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet ee298406a4824f1aa2fa742b7dec1026 from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:47003: session id = 14002a0a4c674b438c78ca5db73637cc-ee298406a4824f1aa2fa742b7dec1026
I20240501 19:57:57.259769 11395 tablet_copy_source_session.cc:205] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.260576 13037 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.261706 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:57.263070 21067 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:57.266165 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.268185 21333 tablet_copy_client.cc:783] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.268440 21333 tablet_copy_client.cc:647] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.277294 21333 tablet_copy_client.cc:523] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.278681 21333 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:57.283604 11355 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:57:57.285082 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.286960 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.290565 11092 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:57.305174 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.307163 21067 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.308122 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.310582 10836 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.312769 10983 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.314188 13037 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:57.315099 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet 2def1cda07c349c69b3abe25804d3d1d with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:57.320493 10983 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:57:57.321184 21067 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:57:57.322855 19925 raft_consensus.cc:979] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: : Instructing follower e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:57.322939 19925 raft_consensus.cc:1067] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer e3c6b84864cb4358a9d7e0f2a65ba83d to start an election
I20240501 19:57:57.323387 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751"
dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
from {username='slave'} at 127.30.1.68:42723
I20240501 19:57:57.323482 13037 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:57.323527 13037 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.324398 13037 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.324760 13037 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:57.325775 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:57.325872 10982 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.326411 13016 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.326743 10982 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:57.327195 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:57.327301 11376 raft_consensus.cc:3032] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:57.327343 11376 raft_consensus.cc:729] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:57.327466 11376 consensus_queue.cc:260] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.327627 11376 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.328565 11376 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:57.329061 12973 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:57:57.329456 12639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a170766f497f491ba9b45f92f2a0e751" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "417972166f4d40ca9b4b6471a90be6af"
I20240501 19:57:57.329592 12639 raft_consensus.cc:3037] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.330641 12639 raft_consensus.cc:2454] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c6b84864cb4358a9d7e0f2a65ba83d in term 2.
I20240501 19:57:57.331318 21067 raft_consensus.cc:2781] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:57.331393 21067 raft_consensus.cc:686] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Becoming Leader. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Running, Role: LEADER
I20240501 19:57:57.331506 21067 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.333618 10838 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72). New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:57.336588 21067 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:57.338243 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2def1cda07c349c69b3abe25804d3d1d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:57.338774 10983 raft_consensus.cc:2932] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:57.339773 12905 raft_consensus.cc:1203] T 19f79cd7d4ec4b4697a2637e6d14a892 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 19:57:57.342908 10833 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 9 to 10, VOTER 14002a0a4c674b438c78ca5db73637cc (127.30.1.70) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:57.353600 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.354794 21555 tablet_replica.cc:330] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:57.354972 21555 raft_consensus.cc:2229] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:57.355132 21555 raft_consensus.cc:2258] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:57.355825 21555 ts_tablet_manager.cc:1875] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:57.358206 21555 ts_tablet_manager.cc:1888] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 19:57:57.358330 21555 log.cc:1196] T 2def1cda07c349c69b3abe25804d3d1d P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/2def1cda07c349c69b3abe25804d3d1d
I20240501 19:57:57.358942 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet 2def1cda07c349c69b3abe25804d3d1d (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:57.381984 12905 raft_consensus.cc:1203] T b4f0699b5bdf44499c87ff1541cd2ad3 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:57.390874 20772 raft_consensus.cc:979] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e: : Instructing follower 417972166f4d40ca9b4b6471a90be6af to start an election
I20240501 19:57:57.390980 20772 raft_consensus.cc:1067] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Signalling peer 417972166f4d40ca9b4b6471a90be6af to start an election
I20240501 19:57:57.391584 12639 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16"
dest_uuid: "417972166f4d40ca9b4b6471a90be6af"
from {username='slave'} at 127.30.1.66:32859
I20240501 19:57:57.391778 12639 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:57.391842 12639 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 3 FOLLOWER]: Advancing to term 4
I20240501 19:57:57.393126 12639 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.394711 12639 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [CANDIDATE]: Term 4 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:57.395198 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "417972166f4d40ca9b4b6471a90be6af" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:57.395325 11113 raft_consensus.cc:3032] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 19:57:57.395365 11113 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:57.395486 11113 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.395654 11113 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 3 FOLLOWER]: Advancing to term 4
I20240501 19:57:57.396634 11113 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417972166f4d40ca9b4b6471a90be6af in term 4.
I20240501 19:57:57.397267 12576 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417972166f4d40ca9b4b6471a90be6af, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:57.397590 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "417972166f4d40ca9b4b6471a90be6af" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
I20240501 19:57:57.397732 13037 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 3 FOLLOWER]: Advancing to term 4
I20240501 19:57:57.398986 13037 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417972166f4d40ca9b4b6471a90be6af in term 4.
I20240501 19:57:57.407831 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 19:57:57.409694 12971 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165): Couldn't send request to peer 14002a0a4c674b438c78ca5db73637cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2def1cda07c349c69b3abe25804d3d1d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.414774 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.423642 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.424598 12621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.428313 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.432404 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.432404 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.437238 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.437687 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.438900 21333 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.439215 21333 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:57.443457 21333 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.165s user 0.026s sys 0.001s
I20240501 19:57:57.443890 21333 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:57.444025 21333 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.444167 21333 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:57.444427 21333 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:57.445753 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-ee298406a4824f1aa2fa742b7dec1026 received from {username='slave'} at 127.30.1.70:47003
I20240501 19:57:57.445819 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-ee298406a4824f1aa2fa742b7dec1026 on tablet ee298406a4824f1aa2fa742b7dec1026 with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:57.448324 12772 raft_consensus.cc:1203] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:57:57.451902 12772 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:57.453462 11245 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240501 19:57:57.454001 12708 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): Couldn't send request to peer 92ecfb9f8f6745efa5aa2e168bafcc4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.454202 17909 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:57.454913 12639 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:57.455359 17909 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:57.457432 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.459066 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.461715 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.465070 21583 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 4 FOLLOWER]: Leader election won for term 4
I20240501 19:57:57.465579 21583 raft_consensus.cc:686] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 4 LEADER]: Becoming Leader. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Running, Role: LEADER
I20240501 19:57:57.472213 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.473685 11244 raft_consensus.cc:1203] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 19:57:57.474053 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.494521 21583 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.497926 21621 ts_tablet_manager.cc:902] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:57.498816 10838 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af reported cstate change: term changed from 3 to 4, leader changed from 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) to 417972166f4d40ca9b4b6471a90be6af (127.30.1.69). New cstate: current_term: 4 leader_uuid: "417972166f4d40ca9b4b6471a90be6af" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 19:57:57.499694 21621 tablet_copy_client.cc:308] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:57.500170 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 219742bd987e4a1990233fa8c972c232 from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:56035)
I20240501 19:57:57.500299 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 219742bd987e4a1990233fa8c972c232 from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:56035: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-219742bd987e4a1990233fa8c972c232
I20240501 19:57:57.501217 11134 tablet_copy_source_session.cc:205] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.502337 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219742bd987e4a1990233fa8c972c232. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.504386 21621 tablet_copy_client.cc:783] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.504539 21621 tablet_copy_client.cc:647] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.505822 21621 tablet_copy_client.cc:523] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.508240 21621 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:57.514890 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.519618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.538370 21621 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.538625 21621 tablet_bootstrap.cc:492] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:57.538765 21621 ts_tablet_manager.cc:1367] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.000s
I20240501 19:57:57.539129 21621 raft_consensus.cc:348] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.539225 21621 raft_consensus.cc:729] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.539337 21621 consensus_queue.cc:260] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.539567 21621 ts_tablet_manager.cc:1398] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:57.539952 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-219742bd987e4a1990233fa8c972c232 received from {username='slave'} at 127.30.1.72:56035
I20240501 19:57:57.540010 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-219742bd987e4a1990233fa8c972c232 on tablet 219742bd987e4a1990233fa8c972c232 with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:57.544582 21633 ts_tablet_manager.cc:902] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:57.544719 21610 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.545462 21633 tablet_copy_client.cc:308] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:57.545744 11245 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.546061 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 19d8d7d255494a49920849d10b800fa7 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e ({username='slave'} at 127.30.1.66:56439)
I20240501 19:57:57.546180 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 19d8d7d255494a49920849d10b800fa7 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e at {username='slave'} at 127.30.1.66:56439: session id = 92ecfb9f8f6745efa5aa2e168bafcc4e-19d8d7d255494a49920849d10b800fa7
I20240501 19:57:57.547030 11395 tablet_copy_source_session.cc:205] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.548035 12639 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.549261 10836 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:57.552186 21633 ts_tablet_manager.cc:1875] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:57.554858 21633 ts_tablet_manager.cc:1888] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 19:57:57.555138 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d8d7d255494a49920849d10b800fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.556672 21633 tablet_copy_client.cc:783] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.556950 21633 tablet_copy_client.cc:647] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.559725 21633 tablet_copy_client.cc:523] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.561324 21633 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:57.579140 13037 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:57.580987 11245 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240501 19:57:57.581015 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 156d2ba7ff2c403db2f63cbae4acfec1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.581601 10983 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:57:57.582074 21553 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:57.582998 21553 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:57:57.585677 21553 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.589798 11245 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.592160 10838 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.593987 10983 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.614284 11245 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:57.615386 12905 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:57.615759 21343 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:57.617496 21343 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.620066 10836 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 5 to 7, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.621129 12904 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:57.623469 10963 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6621e1ff294dc9a474e076873cc72c with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52906
I20240501 19:57:57.623796 21724 tablet_replica.cc:330] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 19:57:57.624056 21724 raft_consensus.cc:2229] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:57.624224 21724 raft_consensus.cc:2258] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:57.624409 11245 consensus_queue.cc:237] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.624825 21724 ts_tablet_manager.cc:1875] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:57.626384 12905 raft_consensus.cc:1261] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 19:57:57.626816 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d6621e1ff294dc9a474e076873cc72c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:57.627086 21343 consensus_queue.cc:1035] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:57.629356 21343 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:57.629760 12905 raft_consensus.cc:2932] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:57.630894 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2d6621e1ff294dc9a474e076873cc72c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:57.631148 21633 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.631522 21633 tablet_bootstrap.cc:492] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap complete.
I20240501 19:57:57.631737 21633 ts_tablet_manager.cc:1367] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.070s user 0.028s sys 0.002s
I20240501 19:57:57.632259 21633 raft_consensus.cc:348] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:57.632422 21633 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.632606 21633 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:57.633496 21633 ts_tablet_manager.cc:1398] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:57.633282 21724 ts_tablet_manager.cc:1888] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 19:57:57.633658 21724 log.cc:1196] T 2d6621e1ff294dc9a474e076873cc72c P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/2d6621e1ff294dc9a474e076873cc72c
I20240501 19:57:57.634013 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-19d8d7d255494a49920849d10b800fa7 received from {username='slave'} at 127.30.1.66:56439
I20240501 19:57:57.634112 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-19d8d7d255494a49920849d10b800fa7 on tablet 19d8d7d255494a49920849d10b800fa7 with peer 92ecfb9f8f6745efa5aa2e168bafcc4e
I20240501 19:57:57.635679 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet 2d6621e1ff294dc9a474e076873cc72c (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:57.636175 10833 catalog_manager.cc:5484] T 2d6621e1ff294dc9a474e076873cc72c P ea26a64181784350bcd3540d0633b219 reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) added. New cstate: current_term: 2 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:57.655967 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.663471 12621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.676668 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.678108 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.681565 12772 raft_consensus.cc:1203] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:57.683601 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.687687 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.703526 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.715893 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.766716 11376 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:57.767071 21666 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:57.775159 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.778709 11113 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44638
I20240501 19:57:57.778821 11113 raft_consensus.cc:595] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:57.789999 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.791905 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.802835 12621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.811380 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.811465 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.813663 21621 ts_tablet_manager.cc:902] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 19:57:57.816824 21621 tablet_copy_client.cc:308] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 19:57:57.817281 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 269047cc3a504d7488747166efce71fc from peer e3c6b84864cb4358a9d7e0f2a65ba83d ({username='slave'} at 127.30.1.72:56035)
I20240501 19:57:57.817364 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 269047cc3a504d7488747166efce71fc from peer e3c6b84864cb4358a9d7e0f2a65ba83d at {username='slave'} at 127.30.1.72:56035: session id = e3c6b84864cb4358a9d7e0f2a65ba83d-269047cc3a504d7488747166efce71fc
I20240501 19:57:57.818408 11134 tablet_copy_source_session.cc:205] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.819165 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269047cc3a504d7488747166efce71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.821182 21621 tablet_copy_client.cc:783] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.821308 21621 tablet_copy_client.cc:647] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.823580 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.827370 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.832242 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.832989 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.852696 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.853230 21621 tablet_copy_client.cc:523] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.854555 21621 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap starting.
I20240501 19:57:57.858408 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.869941 11092 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:57:57.875628 13037 raft_consensus.cc:1203] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:57.878507 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.880472 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.894201 12621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.897547 21621 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:57.897876 21621 tablet_bootstrap.cc:492] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Bootstrap complete.
I20240501 19:57:57.898063 21621 ts_tablet_manager.cc:1367] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.002s
I20240501 19:57:57.898483 21621 raft_consensus.cc:348] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.898669 21621 raft_consensus.cc:729] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c6b84864cb4358a9d7e0f2a65ba83d, State: Initialized, Role: FOLLOWER
I20240501 19:57:57.898835 21621 consensus_queue.cc:260] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 19:57:57.900882 21621 ts_tablet_manager.cc:1398] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:57.901100 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:57.902263 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.902743 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-269047cc3a504d7488747166efce71fc received from {username='slave'} at 127.30.1.72:56035
I20240501 19:57:57.902817 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session e3c6b84864cb4358a9d7e0f2a65ba83d-269047cc3a504d7488747166efce71fc on tablet 269047cc3a504d7488747166efce71fc with peer e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:57.911664 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.921429 12621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.927222 21333 ts_tablet_manager.cc:902] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:57.927628 21333 tablet_copy_client.cc:308] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:57.933269 10983 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:57.933693 21666 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:57.933835 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240501 19:57:57.934355 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:57.934579 21614 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:57:57.943835 12639 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:57.944957 21553 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:57.949553 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:57.950446 11112 raft_consensus.cc:1203] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 19:57:57.964093 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:57.973575 21633 ts_tablet_manager.cc:902] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 19:57:57.974030 21633 tablet_copy_client.cc:308] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 19:57:57.974700 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e ({username='slave'} at 127.30.1.66:41419)
I20240501 19:57:57.974785 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e at {username='slave'} at 127.30.1.66:41419: session id = 92ecfb9f8f6745efa5aa2e168bafcc4e-2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:57.975719 12793 tablet_copy_source_session.cc:205] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:57.976348 21633 ts_tablet_manager.cc:1875] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:57.978514 21633 ts_tablet_manager.cc:1888] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 19:57:57.978772 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:57.981024 21633 tablet_copy_client.cc:783] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:57.981230 21633 tablet_copy_client.cc:647] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:57.983024 13037 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240501 19:57:57.983521 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:57.983683 21614 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:57:57.987892 20867 raft_consensus.cc:979] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e: : Instructing follower 417972166f4d40ca9b4b6471a90be6af to start an election
I20240501 19:57:57.987957 20867 raft_consensus.cc:1067] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Signalling peer 417972166f4d40ca9b4b6471a90be6af to start an election
I20240501 19:57:57.988363 12640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e"
dest_uuid: "417972166f4d40ca9b4b6471a90be6af"
from {username='slave'} at 127.30.1.66:32859
I20240501 19:57:57.988461 12640 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:57.988505 12640 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.989602 12640 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.989935 21633 tablet_copy_client.cc:523] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:57.990012 12640 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [CANDIDATE]: Term 2 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843)
I20240501 19:57:57.991008 11112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "417972166f4d40ca9b4b6471a90be6af" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:57:57.991144 11112 raft_consensus.cc:3032] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:57.991205 11112 raft_consensus.cc:729] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 19:57:57.991339 11112 consensus_queue.cc:260] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.991499 11112 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:57.992362 11112 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417972166f4d40ca9b4b6471a90be6af in term 2.
I20240501 19:57:57.992633 12576 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417972166f4d40ca9b4b6471a90be6af, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters:
I20240501 19:57:57.992835 21614 raft_consensus.cc:2781] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:57.992926 21614 raft_consensus.cc:686] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [term 2 LEADER]: Becoming Leader. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Running, Role: LEADER
I20240501 19:57:57.993041 21614 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:57.991156 21633 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:57.994874 10837 catalog_manager.cc:5484] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af reported cstate change: term changed from 1 to 2, leader changed from 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) to 417972166f4d40ca9b4b6471a90be6af (127.30.1.69). New cstate: current_term: 2 leader_uuid: "417972166f4d40ca9b4b6471a90be6af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:57.998615 21909 ts_tablet_manager.cc:902] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:57:58.007114 21909 tablet_copy_client.cc:308] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 19:57:58.007571 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer ea26a64181784350bcd3540d0633b219 ({username='slave'} at 127.30.1.71:55777)
I20240501 19:57:58.007678 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 156d2ba7ff2c403db2f63cbae4acfec1 from peer ea26a64181784350bcd3540d0633b219 at {username='slave'} at 127.30.1.71:55777: session id = ea26a64181784350bcd3540d0633b219-156d2ba7ff2c403db2f63cbae4acfec1
I20240501 19:57:58.008518 13057 tablet_copy_source_session.cc:205] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:58.009248 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 2def1cda07c349c69b3abe25804d3d1d from peer 14002a0a4c674b438c78ca5db73637cc ({username='slave'} at 127.30.1.70:57641)
I20240501 19:57:58.009339 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 2def1cda07c349c69b3abe25804d3d1d from peer 14002a0a4c674b438c78ca5db73637cc at {username='slave'} at 127.30.1.70:57641: session id = 14002a0a4c674b438c78ca5db73637cc-2def1cda07c349c69b3abe25804d3d1d
I20240501 19:57:58.009382 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d2ba7ff2c403db2f63cbae4acfec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:58.010041 13057 tablet_copy_source_session.cc:205] T 2def1cda07c349c69b3abe25804d3d1d P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:58.010764 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2def1cda07c349c69b3abe25804d3d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:58.012688 21333 tablet_copy_client.cc:783] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:58.012813 21333 tablet_copy_client.cc:647] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:58.014639 21909 tablet_copy_client.cc:783] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:58.014835 21909 tablet_copy_client.cc:647] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:58.016214 21333 tablet_copy_client.cc:523] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:58.017241 21333 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Bootstrap starting.
I20240501 19:57:58.032490 21909 tablet_copy_client.cc:523] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:58.033767 21909 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Bootstrap starting.
I20240501 19:57:58.044195 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49b9c5ebaab42e18fe7bbe8127d569e" candidate_uuid: "417972166f4d40ca9b4b6471a90be6af" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "ea26a64181784350bcd3540d0633b219"
I20240501 19:57:58.044314 12904 raft_consensus.cc:3037] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:58.045253 12904 raft_consensus.cc:2454] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417972166f4d40ca9b4b6471a90be6af in term 2.
I20240501 19:57:58.056677 13037 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:58.057996 12639 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:57:58.058393 21553 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:57:58.062700 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:58.066051 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.071817 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.076587 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.077710 21553 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:58.079252 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.080462 12639 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:58.081871 10836 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 10 to 11, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:58.086243 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd958b083bb430a99130a18f91acdf7 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:58.086931 13037 consensus_queue.cc:237] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
W20240501 19:57:58.088976 12972 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd958b083bb430a99130a18f91acdf7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:58.089005 12639 raft_consensus.cc:1261] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:57:58.089629 21067 consensus_queue.cc:1035] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:57:58.094408 21067 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:58.096772 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6fd958b083bb430a99130a18f91acdf7 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 19:57:58.097200 12639 raft_consensus.cc:2932] T 6fd958b083bb430a99130a18f91acdf7 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:58.097620 10841 catalog_manager.cc:5484] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 11 to 12, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } }
I20240501 19:57:58.099277 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.099365 21909 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:58.099668 21909 tablet_bootstrap.cc:492] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Bootstrap complete.
I20240501 19:57:58.099857 21909 ts_tablet_manager.cc:1367] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.005s
I20240501 19:57:58.100337 21909 raft_consensus.cc:348] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:58.100473 21909 raft_consensus.cc:729] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea26a64181784350bcd3540d0633b219, State: Initialized, Role: FOLLOWER
I20240501 19:57:58.100641 21909 consensus_queue.cc:260] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:58.101052 21909 ts_tablet_manager.cc:1398] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:58.102052 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.102273 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session ea26a64181784350bcd3540d0633b219-156d2ba7ff2c403db2f63cbae4acfec1 received from {username='slave'} at 127.30.1.71:55777
I20240501 19:57:58.102339 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session ea26a64181784350bcd3540d0633b219-156d2ba7ff2c403db2f63cbae4acfec1 on tablet 156d2ba7ff2c403db2f63cbae4acfec1 with peer ea26a64181784350bcd3540d0633b219
I20240501 19:57:58.123669 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.124814 21633 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:58.150997 21333 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:58.151327 21333 tablet_bootstrap.cc:492] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Bootstrap complete.
I20240501 19:57:58.151568 21333 ts_tablet_manager.cc:1367] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Time spent bootstrapping tablet: real 0.134s user 0.024s sys 0.005s
I20240501 19:57:58.161962 21333 raft_consensus.cc:348] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:58.162091 21333 raft_consensus.cc:729] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Initialized, Role: FOLLOWER
I20240501 19:57:58.162254 21333 consensus_queue.cc:260] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:58.162569 21333 ts_tablet_manager.cc:1398] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:57:58.162803 21955 tablet_replica.cc:330] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:58.162941 21955 raft_consensus.cc:2229] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:58.163211 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 14002a0a4c674b438c78ca5db73637cc-2def1cda07c349c69b3abe25804d3d1d received from {username='slave'} at 127.30.1.70:57641
I20240501 19:57:58.163292 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 14002a0a4c674b438c78ca5db73637cc-2def1cda07c349c69b3abe25804d3d1d on tablet 2def1cda07c349c69b3abe25804d3d1d with peer 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:58.163398 21955 raft_consensus.cc:2258] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:58.164240 21955 ts_tablet_manager.cc:1875] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:58.166841 21955 ts_tablet_manager.cc:1888] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 19:57:58.167172 21633 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap complete.
I20240501 19:57:58.167387 21633 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.176s user 0.029s sys 0.000s
I20240501 19:57:58.167868 21633 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:58.168048 21633 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:58.168234 21633 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:58.168947 21633 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:58.169138 21955 log.cc:1196] T 6fd958b083bb430a99130a18f91acdf7 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/6fd958b083bb430a99130a18f91acdf7
I20240501 19:57:58.169410 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-2bc8fbbdc94e49d2b8b639083e083df3 received from {username='slave'} at 127.30.1.66:41419
I20240501 19:57:58.169476 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-2bc8fbbdc94e49d2b8b639083e083df3 on tablet 2bc8fbbdc94e49d2b8b639083e083df3 with peer 92ecfb9f8f6745efa5aa2e168bafcc4e
I20240501 19:57:58.169770 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 6fd958b083bb430a99130a18f91acdf7 (table rebalance_test_table_8 [id=602ecc7b041a4114af2ce5333edad839]) successfully deleted
I20240501 19:57:58.174041 21989 ts_tablet_manager.cc:902] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 19:57:58.174618 21989 tablet_copy_client.cc:308] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 19:57:58.175509 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 2d6621e1ff294dc9a474e076873cc72c from peer 417972166f4d40ca9b4b6471a90be6af ({username='slave'} at 127.30.1.69:47607)
I20240501 19:57:58.175607 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 2d6621e1ff294dc9a474e076873cc72c from peer 417972166f4d40ca9b4b6471a90be6af at {username='slave'} at 127.30.1.69:47607: session id = 417972166f4d40ca9b4b6471a90be6af-2d6621e1ff294dc9a474e076873cc72c
I20240501 19:57:58.176470 11265 tablet_copy_source_session.cc:205] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:58.178553 21989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6621e1ff294dc9a474e076873cc72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:58.180927 21989 tablet_copy_client.cc:783] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:58.181125 21989 tablet_copy_client.cc:647] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:58.190340 21989 tablet_copy_client.cc:523] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:58.191592 21989 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Bootstrap starting.
I20240501 19:57:58.210088 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.211452 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:58.224084 21989 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:58.224398 21989 tablet_bootstrap.cc:492] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Bootstrap complete.
I20240501 19:57:58.224579 21989 ts_tablet_manager.cc:1367] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.000s
I20240501 19:57:58.225065 21989 raft_consensus.cc:348] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:58.225198 21989 raft_consensus.cc:729] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417972166f4d40ca9b4b6471a90be6af, State: Initialized, Role: FOLLOWER
I20240501 19:57:58.225363 21989 consensus_queue.cc:260] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:58.225718 21989 ts_tablet_manager.cc:1398] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:57:58.226446 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session 417972166f4d40ca9b4b6471a90be6af-2d6621e1ff294dc9a474e076873cc72c received from {username='slave'} at 127.30.1.69:47607
I20240501 19:57:58.226514 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session 417972166f4d40ca9b4b6471a90be6af-2d6621e1ff294dc9a474e076873cc72c on tablet 2d6621e1ff294dc9a474e076873cc72c with peer 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:58.232172 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.236869 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.239562 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.266933 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.271165 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.276578 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.339241 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.340891 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.343052 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:58.348047 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.361055 11113 raft_consensus.cc:1203] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240501 19:57:58.362735 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.365140 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.369973 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.371596 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.383416 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:58.384202 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.396720 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.403436 12773 raft_consensus.cc:1203] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240501 19:57:58.403633 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.410822 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.427745 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.429628 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.436161 13036 raft_consensus.cc:1203] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:57:58.437237 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.480798 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58486
I20240501 19:57:58.480933 11376 raft_consensus.cc:595] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:57:58.494071 12639 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.10, Last appended by leader: 9, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:58.496031 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240501 19:57:58.496372 13036 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240501 19:57:58.496785 21614 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 19:57:58.497064 12577 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af -> Peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): Couldn't send request to peer 99348cf9a56645808d00c459fb76a6f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:58.497133 21614 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:57:58.499596 11112 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:58.500474 21583 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:58.500802 21583 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af [term 4 LEADER]: Committing config change with OpId 4.11: config changed from index 9 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:58.503685 10836 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af reported cstate change: config changed from index 9 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New cstate: current_term: 4 leader_uuid: "417972166f4d40ca9b4b6471a90be6af" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:58.506187 13036 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Committing config change with OpId 4.11: config changed from index 9 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:58.506691 11112 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Committing config change with OpId 4.11: config changed from index 9 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:57:58.511698 12904 raft_consensus.cc:1261] T c49b9c5ebaab42e18fe7bbe8127d569e P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Refusing update from remote peer 417972166f4d40ca9b4b6471a90be6af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:58.512607 22175 consensus_queue.cc:1035] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:58.553926 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.561501 11355 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:57:58.578634 21909 ts_tablet_manager.cc:902] T 6fd958b083bb430a99130a18f91acdf7 P ea26a64181784350bcd3540d0633b219: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
F20240501 19:57:58.578809 21909 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.queue_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.queue_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_cpu_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_cpu_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_wall_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_wall_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.queue_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.queue_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_cpu_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_cpu_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_wall_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_wall_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.queue_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.queue_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_cpu_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_cpu_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_wall_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_wall_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.queue_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.queue_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_cpu_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_cpu_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_wall_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_wall_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.queue_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.queue_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.run_cpu_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.run_cpu_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.run_wall_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.run_wall_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.queue_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.queue_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_cpu_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_cpu_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_wall_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_wall_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.queue_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.queue_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_cpu_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_cpu_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_wall_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_wall_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us) (tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.queue_time_us, tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.queue_time_us) (tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.run_cpu_time_us, tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.run_cpu_time_us) (tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.run_wall_time_us, tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.run_wall_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.queue_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.queue_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_cpu_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_cpu_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_wall_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_wall_time_us) (tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.queue_time_us, tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.queue_time_us) (tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.run_cpu_time_us, tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.run_cpu_time_us) (tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.run_wall_time_us, tablet-download-pool-b4f0699b5bdf44499c87ff1541cd2ad3.run_wall_time_us) (tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.queue_time_us, tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.queue_time_us) (tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.run_cpu_time_us, tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.run_cpu_time_us) (tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.run_wall_time_us, tablet-download-pool-c3ee13ec9bb1468f90cfc184455f9fff.run_wall_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.queue_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.queue_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_cpu_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_cpu_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_wall_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_wall_time_us) (tablet-download-pool-d5faf889870246b78b82e19c041dc119.queue_time_us, tablet-download-pool-d5faf889870246b78b82e19c041dc119.queue_time_us) (tablet-download-pool-d5faf889870246b78b82e19c041dc119.run_cpu_time_us, tablet-download-pool-d5faf889870246b78b82e19c041dc119.run_cpu_time_us) (tablet-download-pool-d5faf889870246b78b82e19c041dc119.run_wall_time_us, tablet-download-pool-d5faf889870246b78b82e19c041dc119.run_wall_time_us) (tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.queue_time_us, tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.queue_time_us) (tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.run_cpu_time_us, tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.run_cpu_time_us) (tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.run_wall_time_us, tablet-download-pool-ee298406a4824f1aa2fa742b7dec1026.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 19:57:58.579795 12904 raft_consensus.cc:1203] T 156d2ba7ff2c403db2f63cbae4acfec1 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
@ 0x7fe255960dcd google::LogMessage::Fail() at ??:0
I20240501 19:57:58.585546 12639 raft_consensus.cc:1203] T 2d6621e1ff294dc9a474e076873cc72c P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
@ 0x7fe255964b93 google::LogMessage::SendToLog() at ??:0
I20240501 19:57:58.589612 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fe2559607cc google::LogMessage::Flush() at ??:0
I20240501 19:57:58.594040 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.602645 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7fe255961f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 19:57:58.618824 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.621313 12753 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:57:58.625597 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.627239 21179 raft_consensus.cc:979] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: : Instructing follower 14002a0a4c674b438c78ca5db73637cc to start an election
I20240501 19:57:58.627316 21179 raft_consensus.cc:1067] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer 14002a0a4c674b438c78ca5db73637cc to start an election
I20240501 19:57:58.628021 12773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026"
dest_uuid: "14002a0a4c674b438c78ca5db73637cc"
from {username='slave'} at 127.30.1.68:59057
I20240501 19:57:58.628134 12773 raft_consensus.cc:482] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:58.628188 12773 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fe256dca044 kudu::TraceMetrics::InternName() at ??:0
I20240501 19:57:58.629313 12773 raft_consensus.cc:504] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:58.629810 12773 leader_election.cc:288] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 2 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843)
I20240501 19:57:58.630743 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 19:57:58.630877 11376 raft_consensus.cc:3032] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:58.630925 11376 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 19:57:58.631050 11376 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:58.631219 11376 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fe256daef98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 19:57:58.632138 11376 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14002a0a4c674b438c78ca5db73637cc in term 2.
I20240501 19:57:58.632592 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 19:57:58.632709 10983 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:58.633646 10983 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14002a0a4c674b438c78ca5db73637cc in term 2.
I20240501 19:57:58.634150 12709 leader_election.cc:302] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14002a0a4c674b438c78ca5db73637cc, 1679f84270414f67ac67c882220e1472, 99348cf9a56645808d00c459fb76a6f8; no voters:
I20240501 19:57:58.634303 17909 raft_consensus.cc:2781] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:58.634424 17909 raft_consensus.cc:686] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Becoming Leader. State: Replica: 14002a0a4c674b438c78ca5db73637cc, State: Running, Role: LEADER
I20240501 19:57:58.634564 17909 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:58.636831 10835 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to 14002a0a4c674b438c78ca5db73637cc (127.30.1.70). New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 19:57:58.646838 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:58.647392 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.651624 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.659545 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.682845 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.691881 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.692386 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.697633 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.701105 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7fe256dacefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 19:57:58.709831 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.713079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.714202 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:58.715389 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fe25995f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe259968a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 19:57:58.738499 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.739717 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:58.740017 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:58.746778 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.755925 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fe2599e976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe2599fd4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 19:57:58.768419 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.770006 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7fe2599e8226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fe2599f7c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560f8449675c std::function<>::operator()() at ??:0
I20240501 19:57:58.781525 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee298406a4824f1aa2fa742b7dec1026" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ea26a64181784350bcd3540d0633b219"
I20240501 19:57:58.781646 12904 raft_consensus.cc:3037] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:58.782657 12904 raft_consensus.cc:2454] T ee298406a4824f1aa2fa742b7dec1026 P ea26a64181784350bcd3540d0633b219 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14002a0a4c674b438c78ca5db73637cc in term 2.
@ 0x7fe256db1e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 19:57:58.785152 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.785199 12885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:57:58.787595 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.818614 11376 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
@ 0x7fe256db26ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
W20240501 19:57:58.825659 11311 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): Couldn't send request to peer 92ecfb9f8f6745efa5aa2e168bafcc4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
@ 0x7fe256db403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:58.826197 12639 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:58.826560 22287 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x560f8449675c std::function<>::operator()() at ??:0
I20240501 19:57:58.828123 12773 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:58.828490 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:58.828474 22287 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7fe256da1807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2575df6db start_thread at ??:0
@ 0x7fe254b1771f clone at ??:0
I20240501 19:57:58.874202 22287 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.876708 12639 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.877709 12772 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.880544 10839 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 417972166f4d40ca9b4b6471a90be6af reported cstate change: config changed from index 7 to 8, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.905253 22347 ts_tablet_manager.cc:902] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Initiating tablet copy from peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:57:58.905768 22347 tablet_copy_client.cc:308] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.69:38171
I20240501 19:57:58.906731 12659 tablet_copy_service.cc:140] P 417972166f4d40ca9b4b6471a90be6af: Received BeginTabletCopySession request for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer 99348cf9a56645808d00c459fb76a6f8 ({username='slave'} at 127.30.1.65:43991)
I20240501 19:57:58.906855 12659 tablet_copy_service.cc:161] P 417972166f4d40ca9b4b6471a90be6af: Beginning new tablet copy session on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 from peer 99348cf9a56645808d00c459fb76a6f8 at {username='slave'} at 127.30.1.65:43991: session id = 99348cf9a56645808d00c459fb76a6f8-dbc6ed9f4bf34b1caa462aa9d85c7a16
I20240501 19:57:58.908174 12659 tablet_copy_source_session.cc:205] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 417972166f4d40ca9b4b6471a90be6af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:58.909488 22347 ts_tablet_manager.cc:1875] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:58.911733 12639 consensus_queue.cc:237] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:58.915468 22347 ts_tablet_manager.cc:1888] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240501 19:57:58.915637 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:58.916543 22347 tablet_copy_client.cc:783] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:58.916664 22347 tablet_copy_client.cc:647] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:58.918362 22347 tablet_copy_client.cc:523] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:58.919569 22347 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 19:57:58.932626 10983 consensus_queue.cc:237] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
W20240501 19:57:58.934705 10919 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5faf889870246b78b82e19c041dc119. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:58.934826 12772 raft_consensus.cc:1261] T d5faf889870246b78b82e19c041dc119 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:57:58.935972 20663 consensus_queue.cc:1035] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:58.943215 12772 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:58.944557 12639 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:57:58.944591 11113 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:57:58.944981 17909 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:57:58.945542 17909 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:57:58.947362 17909 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.949265 12639 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.949961 11113 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.951233 10841 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 19:57:58.954346 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc8fbbdc94e49d2b8b639083e083df3 with delete_type TABLET_DATA_TOMBSTONED (TS 4456bc4917c644f8b69b4f5bfb8e1166 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44156
I20240501 19:57:58.960019 22412 tablet_replica.cc:330] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: stopping tablet replica
I20240501 19:57:58.960140 22412 raft_consensus.cc:2229] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:58.960682 22412 raft_consensus.cc:2258] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:58.961196 22412 ts_tablet_manager.cc:1875] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:58.964144 22412 ts_tablet_manager.cc:1888] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:57:58.964202 22412 log.cc:1196] T 2bc8fbbdc94e49d2b8b639083e083df3 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-2/wal/wals/2bc8fbbdc94e49d2b8b639083e083df3
I20240501 19:57:58.964720 10826 catalog_manager.cc:4830] TS 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): tablet 2bc8fbbdc94e49d2b8b639083e083df3 (table rebalance_test_table_9 [id=7b7b68f5ef74432e98060f6b9cfe50ce]) successfully deleted
I20240501 19:57:59.005187 22347 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:59.005471 22347 tablet_bootstrap.cc:492] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap complete.
I20240501 19:57:59.005631 22347 ts_tablet_manager.cc:1367] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.086s user 0.030s sys 0.000s
I20240501 19:57:59.006016 22347 raft_consensus.cc:348] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:59.006115 22347 raft_consensus.cc:729] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 19:57:59.006300 22347 consensus_queue.cc:260] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 4.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:57:59.006636 22347 ts_tablet_manager.cc:1398] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:59.007236 12659 tablet_copy_service.cc:342] P 417972166f4d40ca9b4b6471a90be6af: Request end of tablet copy session 99348cf9a56645808d00c459fb76a6f8-dbc6ed9f4bf34b1caa462aa9d85c7a16 received from {username='slave'} at 127.30.1.65:43991
I20240501 19:57:59.007316 12659 tablet_copy_service.cc:433] P 417972166f4d40ca9b4b6471a90be6af: ending tablet copy session 99348cf9a56645808d00c459fb76a6f8-dbc6ed9f4bf34b1caa462aa9d85c7a16 on tablet dbc6ed9f4bf34b1caa462aa9d85c7a16 with peer 99348cf9a56645808d00c459fb76a6f8
I20240501 19:57:59.022722 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:59.023159 22407 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:59.086211 11376 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:57:59.086643 22407 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:57:59.089639 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.090561 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.093171 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.096789 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.108214 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:57:59.122428 12576 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.122483 11180 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.122568 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.122558 12576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.122988 12972 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.123090 12708 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.123106 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.123176 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:59.123579 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:57:59.123770 11311 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.123868 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.124245 10918 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.124347 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.124677 11049 connection.cc:535] client connection to 127.30.1.71:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.124783 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.127645 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.130008 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.130838 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.130900 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.132345 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.132403 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.132436 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:57:59.132755 12576 consensus_peers.cc:485] T c49b9c5ebaab42e18fe7bbe8127d569e P 417972166f4d40ca9b4b6471a90be6af -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.139250 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:59.141557 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.141614 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.144043 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.145021 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:57:59.155869 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.156862 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.156947 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.171757 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.175034 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.180408 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.181316 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.184319 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.237562 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.245038 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.249611 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.251402 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:57:59.259251 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.260125 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.267093 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.270946 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.291116 13036 consensus_queue.cc:237] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:57:59.292634 12639 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:59.293207 21553 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:59.294310 10983 raft_consensus.cc:1261] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:57:59.294617 21553 consensus_queue.cc:1035] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:57:59.296361 21553 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:59.296644 10983 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:59.299098 10832 catalog_manager.cc:5484] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:59.301818 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet a170766f497f491ba9b45f92f2a0e751 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34106
I20240501 19:57:59.304083 22586 tablet_replica.cc:330] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 19:57:59.304244 22592 ts_tablet_manager.cc:902] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 19:57:59.304265 22586 raft_consensus.cc:2229] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:59.304365 22586 raft_consensus.cc:2258] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:59.304859 22592 tablet_copy_client.cc:308] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 19:57:59.304946 22586 ts_tablet_manager.cc:1875] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:59.305291 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e ({username='slave'} at 127.30.1.66:56439)
I20240501 19:57:59.305372 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 6edb640b8cc04d20b0815b1beb45d9c8 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e at {username='slave'} at 127.30.1.66:56439: session id = 92ecfb9f8f6745efa5aa2e168bafcc4e-6edb640b8cc04d20b0815b1beb45d9c8
I20240501 19:57:59.306241 11395 tablet_copy_source_session.cc:205] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:59.306893 22592 ts_tablet_manager.cc:1875] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:57:59.307580 22586 ts_tablet_manager.cc:1888] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 19:57:59.307708 22586 log.cc:1196] T a170766f497f491ba9b45f92f2a0e751 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/a170766f497f491ba9b45f92f2a0e751
I20240501 19:57:59.308292 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet a170766f497f491ba9b45f92f2a0e751 (table rebalance_test_table_3 [id=b6f2f097e1e645e19a7696601cf4fb10]) successfully deleted
I20240501 19:57:59.309624 22592 ts_tablet_manager.cc:1888] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 19:57:59.309829 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edb640b8cc04d20b0815b1beb45d9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:59.310968 22592 tablet_copy_client.cc:783] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:59.311167 22592 tablet_copy_client.cc:647] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:59.312139 22592 tablet_copy_client.cc:523] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:59.314627 22592 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 19:57:59.315392 12640 raft_consensus.cc:2932] T a170766f497f491ba9b45f92f2a0e751 P 417972166f4d40ca9b4b6471a90be6af [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 19:57:59.336974 10983 raft_consensus.cc:1203] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 4.10->[4.11-4.11] Dedup: 4.11->[]
I20240501 19:57:59.345927 22592 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:59.346163 22592 tablet_bootstrap.cc:492] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap complete.
I20240501 19:57:59.346338 22592 ts_tablet_manager.cc:1367] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.005s
I20240501 19:57:59.346695 22592 raft_consensus.cc:348] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:59.346794 22592 raft_consensus.cc:729] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 19:57:59.346931 22592 consensus_queue.cc:260] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 19:57:59.347858 22592 ts_tablet_manager.cc:1398] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:59.349624 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-6edb640b8cc04d20b0815b1beb45d9c8 received from {username='slave'} at 127.30.1.66:56439
I20240501 19:57:59.349696 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-6edb640b8cc04d20b0815b1beb45d9c8 on tablet 6edb640b8cc04d20b0815b1beb45d9c8 with peer 92ecfb9f8f6745efa5aa2e168bafcc4e
I20240501 19:57:59.399459 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.399536 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.399677 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.405251 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.408632 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.415932 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.416972 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.452322 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.452633 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:57:59.462996 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.475078 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:57:59.487596 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.487629 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.489114 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.497040 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.499774 12621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.509558 22711 ts_tablet_manager.cc:902] T d5faf889870246b78b82e19c041dc119 P 417972166f4d40ca9b4b6471a90be6af: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
F20240501 19:57:59.509789 22711 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.queue_time_us, tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.queue_time_us) (tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.run_cpu_time_us, tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.run_cpu_time_us) (tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.run_wall_time_us, tablet-download-pool-25ea8d7ab1834abfa34ff2357aa554b0.run_wall_time_us) (tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.queue_time_us, tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.queue_time_us) (tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.run_cpu_time_us, tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.run_cpu_time_us) (tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.run_wall_time_us, tablet-download-pool-26b12620ad9a404d9ab70e8506ad1a39.run_wall_time_us) (tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.queue_time_us, tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.queue_time_us) (tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.run_cpu_time_us, tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.run_cpu_time_us) (tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.run_wall_time_us, tablet-download-pool-2bc8fbbdc94e49d2b8b639083e083df3.run_wall_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.queue_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.queue_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_cpu_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_cpu_time_us) (tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_wall_time_us, tablet-download-pool-2d6621e1ff294dc9a474e076873cc72c.run_wall_time_us) (tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.queue_time_us, tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.queue_time_us) (tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.run_cpu_time_us, tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.run_cpu_time_us) (tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.run_wall_time_us, tablet-download-pool-318eb8fd9972435a8e51f63bac077f09.run_wall_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.queue_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.queue_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_cpu_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_cpu_time_us) (tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_wall_time_us, tablet-download-pool-5d032fb77ac944629a45b808cc76c2f1.run_wall_time_us) (tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.queue_time_us, tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.queue_time_us) (tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.run_cpu_time_us, tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.run_cpu_time_us) (tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.run_wall_time_us, tablet-download-pool-6edb640b8cc04d20b0815b1beb45d9c8.run_wall_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_cpu_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_cpu_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_wall_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_wall_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.queue_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.queue_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_cpu_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_cpu_time_us) (tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_wall_time_us, tablet-download-pool-8fc8c5aab87341e3b79a8e121e6db01e.run_wall_time_us) (tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.queue_time_us, tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.queue_time_us) (tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.run_cpu_time_us, tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.run_cpu_time_us) (tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.run_wall_time_us, tablet-download-pool-95b8c1a34cc94d29986289939b2159bb.run_wall_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.queue_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.queue_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_cpu_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_cpu_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_wall_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_wall_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.queue_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.queue_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_cpu_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_cpu_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_wall_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_wall_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.queue_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.queue_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_cpu_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_cpu_time_us) (tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_wall_time_us, tablet-download-pool-c49b9c5ebaab42e18fe7bbe8127d569e.run_wall_time_us) (tablet-download-pool-d5faf889870246b78b82e19c041dc119.queue_time_us, tablet-download-pool-d5faf889870246b78b82e19c041dc119.queue_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.queue_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.queue_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_cpu_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_cpu_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_wall_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fca9d605dcd google::LogMessage::Fail() at ??:0
@ 0x7fca9d609b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fca9d6057cc google::LogMessage::Flush() at ??:0
@ 0x7fca9d606f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fca9ea6f044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fca9ea53f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fca9ea51efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fcaa160477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fcaa160da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fcaa168e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fcaa16a24f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 19:57:59.554705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fcaa168d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 19:57:59.555972 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7fcaa169cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563e4359e75c std::function<>::operator()() at ??:0
@ 0x7fca9ea56e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 19:57:59.562114 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fca9ea576ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fca9ea5903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563e4359e75c std::function<>::operator()() at ??:0
@ 0x7fca9ea46807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fca9f2846db start_thread at ??:0
@ 0x7fca9c7bc71f clone at ??:0
I20240501 19:57:59.585734 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.587345 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.599769 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.663448 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.667933 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.668082 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.678792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.679736 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.699045 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:59.796993 11312 connection.cc:535] client connection to 127.30.1.69:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.797117 11181 connection.cc:535] client connection to 127.30.1.69:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.797272 10919 connection.cc:535] client connection to 127.30.1.69:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.798612 12973 connection.cc:535] client connection to 127.30.1.69:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.798687 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.798751 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.799126 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:57:59.799198 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.799515 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.799616 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.799687 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.799755 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.860833 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.863842 11113 raft_consensus.cc:1203] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240501 19:57:59.899420 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:59.909145 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.911691 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.927799 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.928406 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.928702 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.932713 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.932974 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.938659 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:57:59.940153 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:59.946172 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:57:59.953676 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:57:59.955013 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:57:59.966390 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:57:59.971882 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.985109 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:57:59.985368 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:57:59.992549 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.001474 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.001688 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:00.009886 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.012293 21908 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:00.012401 21908 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:00.012838 21908 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:58:00.016497 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:00.017488 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:00.017930 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.017992 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:00.018193 21908 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:00.025557 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.027882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.034152 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.047711 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.056780 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:00.057790 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:58:00.079329 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.081432 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.141528 13086 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:57:43.449904 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.450176 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.454568 13092 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:57:44.286142 13092 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:57:44.763478 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.763597 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.763687 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.763995 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.764042 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.764099 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.764134 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.764166 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.764202 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.764231 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.764262 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.764292 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.764356 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.764392 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.764425 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.764467 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:44.979517 13092 rebalancer_tool.cc:1074] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:45.075138 13092 rebalancer_tool.cc:1074] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:45.145525 13092 rebalancer_tool.cc:1074] tablet 95b8c1a34cc94d29986289939b2159bb: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:45.154304 13092 rebalancer_tool.cc:1084] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move ignored: Illegal state: tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: replica 417972166f4d40ca9b4b6471a90be6af is already present
I20240501 19:57:45.154383 13092 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:45.394546 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.394644 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.394699 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.394762 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.394802 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.394852 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.394896 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.394937 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.394976 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.395056 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.395093 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.395133 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.395177 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.395205 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.395244 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.395283 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.395342 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.395380 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.395429 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.395488 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:45.522033 13092 rebalancer_tool.cc:1084] tablet 318eb8fd9972435a8e51f63bac077f09: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 19:57:45.522132 13092 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:46.052611 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.052701 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.052805 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.052870 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.066016 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.066109 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.066147 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.066175 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.066205 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.066236 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.066264 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.066320 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.066349 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.066393 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.066447 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.066581 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.066637 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.066663 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.066687 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.066716 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.066751 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.066777 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.091754 13092 rebalancer_tool.cc:1084] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move ignored: Illegal state: tablet 26b12620ad9a404d9ab70e8506ad1a39: replica 417972166f4d40ca9b4b6471a90be6af is already present
I20240501 19:57:46.091817 13092 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:46.608027 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.611654 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.611789 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.613746 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.614157 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.614243 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.617082 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.617162 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.617204 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.617254 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.617316 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.617446 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.617523 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.617566 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:46.675309 13092 rebalancer_tool.cc:1074] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:46.944358 13092 rebalancer_tool.cc:1074] tablet b4f0699b5bdf44499c87ff1541cd2ad3: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:47.001106 13092 rebalancer_tool.cc:1084] tablet 54d2db0f85ac4a90a1df779f89398463: '1679f84270414f67ac67c882220e1472' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 19:57:47.001188 13092 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:47.437901 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.438015 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.438066 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.438130 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.438169 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.438241 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.438293 13092 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:47.480154 13092 rebalancer_tool.cc:1074] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:47.533078 13092 rebalancer_tool.cc:1074] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:47.565377 13092 rebalancer_tool.cc:1074] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:47.577649 13092 rebalancer_tool.cc:1074] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:50.042659 13092 rebalancer_tool.cc:1136] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:50.056471 13092 rebalancer_tool.cc:1136] tablet 95b8c1a34cc94d29986289939b2159bb: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:50.073884 13092 rebalancer_tool.cc:1136] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:50.625927 13092 rebalancer_tool.cc:1136] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:50.643539 13092 rebalancer_tool.cc:1074] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:50.697412 13092 rebalancer_tool.cc:1074] tablet 2def1cda07c349c69b3abe25804d3d1d: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:51.578238 13092 rebalancer_tool.cc:1136] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:51.589709 13092 rebalancer_tool.cc:1136] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:51.597908 13092 rebalancer_tool.cc:1136] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:51.615749 13092 rebalancer_tool.cc:1136] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:51.632863 13092 rebalancer_tool.cc:1136] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:51.642061 13092 rebalancer_tool.cc:1136] tablet b4f0699b5bdf44499c87ff1541cd2ad3: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:51.652396 13092 rebalancer_tool.cc:1074] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:51.669263 13092 rebalancer_tool.cc:1074] tablet 9fcd6776aebf47c99f05850de83b36e6: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:51.698078 13092 rebalancer_tool.cc:1074] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:51.797224 13092 rebalancer_tool.cc:1074] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:51.853907 13092 rebalancer_tool.cc:1136] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:53.834281 13092 rebalancer_tool.cc:1074] tablet 6fd958b083bb430a99130a18f91acdf7: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:54.238631 13092 rebalancer_tool.cc:1136] tablet 9fcd6776aebf47c99f05850de83b36e6: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:54.270568 13092 rebalancer_tool.cc:1136] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.485103 13092 rebalancer_tool.cc:1136] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:54.947624 13092 rebalancer_tool.cc:1074] tablet c49b9c5ebaab42e18fe7bbe8127d569e: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.975406 13092 rebalancer_tool.cc:1074] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:5
I20240501 19:58:00.146027 13087 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:57:43.527642 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.527863 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.530982 13093 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:57:44.603351 13093 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:57:45.015959 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.016072 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.016227 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.016295 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.016342 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.016379 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.016427 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.016471 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.016511 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.016554 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.016599 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.016662 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.016707 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.016749 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.016791 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.016834 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:45.078168 13093 rebalancer_tool.cc:1074] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '1679f84270414f67ac67c882220e1472' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:45.166796 13093 rebalancer_tool.cc:1074] tablet c49b9c5ebaab42e18fe7bbe8127d569e: '99348cf9a56645808d00c459fb76a6f8' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:45.302114 13093 rebalancer_tool.cc:1074] tablet 9fcd6776aebf47c99f05850de83b36e6: '4456bc4917c644f8b69b4f5bfb8e1166' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:45.399609 13093 rebalancer_tool.cc:1074] tablet 6fd958b083bb430a99130a18f91acdf7: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:45.407106 13093 rebalancer_tool.cc:1084] tablet 95b8c1a34cc94d29986289939b2159bb: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: tablet 95b8c1a34cc94d29986289939b2159bb: replica 14002a0a4c674b438c78ca5db73637cc is already present
I20240501 19:57:45.407194 13093 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:45.751435 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.751565 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.751648 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.751706 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.751746 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.751785 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.751883 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.751932 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.751978 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.752022 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.752102 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.752147 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.752189 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.752234 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.752302 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.752341 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.752386 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.752426 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.752466 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.752506 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:45.900359 13093 rebalancer_tool.cc:1074] tablet 5d032fb77ac944629a45b808cc76c2f1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:46.078088 13093 rebalancer_tool.cc:1074] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:46.123013 13093 rebalancer_tool.cc:1084] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: tablet 2bc8fbbdc94e49d2b8b639083e083df3: replica 14002a0a4c674b438c78ca5db73637cc is already present
I20240501 19:57:46.123085 13093 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:46.642534 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.642769 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.643464 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.643580 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.643630 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.643671 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.643712 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.643743 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.643770 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.643795 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.643823 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.643863 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.643915 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.643955 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.643996 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:46.660835 13093 rebalancer_tool.cc:1074] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:46.826823 13093 rebalancer_tool.cc:1074] tablet d5faf889870246b78b82e19c041dc119: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:46.993533 13093 rebalancer_tool.cc:1074] tablet 54d2db0f85ac4a90a1df779f89398463: '1679f84270414f67ac67c882220e1472' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:47.005705 13093 rebalancer_tool.cc:1074] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:47.015040 13093 rebalancer_tool.cc:1084] tablet b4f0699b5bdf44499c87ff1541cd2ad3: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: tablet b4f0699b5bdf44499c87ff1541cd2ad3: replica 14002a0a4c674b438c78ca5db73637cc is already present
I20240501 19:57:47.015110 13093 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:47.473798 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.473948 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.474016 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.474103 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.474161 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.474198 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.474243 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.474273 13093 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:47.493093 13093 rebalancer_tool.cc:1074] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:47.528503 13093 rebalancer_tool.cc:1074] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:47.535233 13093 rebalancer_tool.cc:1074] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:50.080384 13093 rebalancer_tool.cc:1136] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:51.215523 13093 rebalancer_tool.cc:1136] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:51.971115 13093 rebalancer_tool.cc:1136] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:52.429172 13093 rebalancer_tool.cc:1123] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '1679f84270414f67ac67c882220e1472' -> '14002a0a4c674b438c78ca5db73637cc' move is abandoned: Not found: Server with UUID 1679f84270414f67ac67c882220e1472 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 19:57:52.916663 13093 rebalancer_tool.cc:1123] tablet 5d032fb77ac944629a45b808cc76c2f1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '417972166f4d40ca9b4b6471a90be6af' move is abandoned: Not found: Server with UUID 92ecfb9f8f6745efa5aa2e168bafcc4e not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } }
I20240501 19:57:53.005365 13093 rebalancer_tool.cc:1074] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:53.030720 13093 rebalancer_tool.cc:1074] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:53.481967 13093 rebalancer_tool.cc:1136] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:53.506227 13093 rebalancer_tool.cc:1136] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:53.519516 13093 rebalancer_tool.cc:1136] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:53.580618 13093 rebalancer_tool.cc:1136] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:53.690764 13093 rebalancer_tool.cc:1136] tablet 54d2db0f85ac4a90a1df779f89398463: '1679f84270414f67ac67c882220e1472' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:53.700389 13093 rebalancer_tool.cc:1136] tablet d5faf889870246b78b82e19c041dc119: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:53.729475 13093 rebalancer_tool.cc:1136] tablet c49b9c5ebaab42e18fe7bbe8127d569e: '99348cf9a56645808d00c459fb76a6f8' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:53.750543 13093 rebalancer_tool.cc:1136] tablet 9fcd6776aebf47c99f05850de83b36e6: '4456bc4917c644f8b69b4f5bfb8e1166' -> '417972166f4d40ca9b4b6471a90be6af' move completed: OK
I20240501 19:57:53.771232 13093 rebalancer_tool.cc:1136] tablet 6fd958b083bb430a99130a18f91acdf7: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:53.821080 13093 rebalancer_tool.cc:1074] tablet 318eb8fd9972435a8e51f63bac077f09: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:53.844069 13093 rebalancer_tool.cc:1074] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:53.855162 13093 rebalancer_tool.cc:1074] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:53.864357 13093 rebalancer_tool.cc:1074] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:53.877316 13093 rebalancer_tool.cc:1136] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:53.888482 13093 rebalancer_tool.cc:1136] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:53.906962 13093 rebalancer_tool.cc:1136] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.658985 13093 rebalancer_tool.cc:1136] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.676641 13093 rebalancer_tool.cc:1074] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.684798 13093 rebalancer_tool.cc:1074] tablet 95b8c1a34cc94d29986289939b2159bb: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.705811 13093 rebalancer_tool.cc:1074] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.719224 13093 rebalancer_tool.cc:1136] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:56.055483 13093 rebalancer_tool.cc:1136] tablet 318eb8fd9972435a8e51f63bac077f09: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:56.062296 13093 rebalancer_tool.cc:1136] tablet 95b8c1a34cc94d29986289939b2159bb: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
W20240501 19:57:59.796433 22682 connection.cc:535] client connection to 127.30.1.69:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:59.796640 22732 ksck_remote.cc:304] Couldn't fetch quiescing info from tablet server 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:59.805881 13093 rebalancer_tool.cc:1123] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:57:59.805943 13093 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): unacceptable health status UNAVAILABLE
I20240501 19:58:00.190464 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.194391 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.200685 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.205284 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.205730 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.208142 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.252569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.258518 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.260228 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.260859 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.276721 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.284298 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.362876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.365288 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.365410 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.365518 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.366511 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.375977 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.455653 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:00.459278 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.465165 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:00.468004 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:00.473930 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.478014 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.478049 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.497339 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.580663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.585628 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.590718 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.592350 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.602933 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.618022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.618505 23060 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:00.618604 23060 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:58:00.619057 23060 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:58:00.619052 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.620596 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:58:00.621048 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:58:00.621876 12973 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e3c6b84864cb4358a9d7e0f2a65ba83d; no voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8
I20240501 19:58:00.622444 23060 raft_consensus.cc:2726] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 19:58:00.623018 12973 leader_election.cc:334] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.628459 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.635397 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.642967 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.652894 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.663185 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.734692 10963 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.735738 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.736371 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.737601 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.738993 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.760238 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.820880 21908 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:00.821012 21908 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:58:00.821521 21908 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 19:58:00.822031 10983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 19:58:00.822326 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" is_pre_election: true
W20240501 19:58:00.822441 11050 leader_election.cc:334] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.822459 13036 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 4.
I20240501 19:58:00.823606 11050 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters: 417972166f4d40ca9b4b6471a90be6af, 99348cf9a56645808d00c459fb76a6f8
I20240501 19:58:00.823830 21908 raft_consensus.cc:2726] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 19:58:00.837208 22391 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:00.837313 22391 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:58:00.837792 22391 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:58:00.838202 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" is_pre_election: true
I20240501 19:58:00.838213 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" is_pre_election: true
I20240501 19:58:00.838325 11113 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 4.
I20240501 19:58:00.838325 13036 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 4.
I20240501 19:58:00.838749 10918 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:58:00.838943 22391 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 19:58:00.838997 22391 raft_consensus.cc:482] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:00.839046 22391 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 4 FOLLOWER]: Advancing to term 5
W20240501 19:58:00.840129 10919 leader_election.cc:334] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.840147 22391 raft_consensus.cc:504] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:58:00.840796 22391 leader_election.cc:288] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 election: Requested vote from peers 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 19:58:00.840976 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
I20240501 19:58:00.841077 11113 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 4 FOLLOWER]: Advancing to term 5
I20240501 19:58:00.841312 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc6ed9f4bf34b1caa462aa9d85c7a16" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d"
I20240501 19:58:00.841411 13036 raft_consensus.cc:3037] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 4 FOLLOWER]: Advancing to term 5
I20240501 19:58:00.842036 11113 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 5.
I20240501 19:58:00.842316 13036 raft_consensus.cc:2454] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 5.
I20240501 19:58:00.842535 10918 leader_election.cc:302] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e, 99348cf9a56645808d00c459fb76a6f8, e3c6b84864cb4358a9d7e0f2a65ba83d; no voters:
I20240501 19:58:00.843566 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:00.845724 10919 leader_election.cc:334] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:00.846745 22391 raft_consensus.cc:2781] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 19:58:00.846894 22391 raft_consensus.cc:686] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 5 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 19:58:00.846954 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:00.847041 22391 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 11, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 19:58:00.847287 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.850605 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.862354 10834 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 4 to 5, leader changed from 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 5 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 19:58:00.863246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:00.875546 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.943619 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.944939 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:00.946408 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:00.967726 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:00.967985 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:00.987099 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.009498 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.017787 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.017860 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.024003 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.025070 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.027277 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.050794 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:01.054695 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:01.074903 11113 consensus_queue.cc:237] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 19:58:01.076026 11376 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:58:01.076078 12773 raft_consensus.cc:1261] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:58:01.076421 23421 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:58:01.081738 23421 consensus_queue.cc:1035] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:58:01.083926 21908 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:58:01.084661 11376 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:58:01.085070 12773 raft_consensus.cc:2932] T 19bfb955b9c94fdb8e0381e5b1855248 P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:58:01.087441 10837 catalog_manager.cc:5484] T 19bfb955b9c94fdb8e0381e5b1855248 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 9 to 10, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) evicted. New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 19:58:01.090448 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bfb955b9c94fdb8e0381e5b1855248 with delete_type TABLET_DATA_TOMBSTONED (TS e3c6b84864cb4358a9d7e0f2a65ba83d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38180
I20240501 19:58:01.090843 23426 tablet_replica.cc:330] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: stopping tablet replica
I20240501 19:58:01.091104 23426 raft_consensus.cc:2229] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:58:01.091269 23426 raft_consensus.cc:2258] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:58:01.091954 23426 ts_tablet_manager.cc:1875] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:58:01.093747 23426 ts_tablet_manager.cc:1888] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 19:58:01.093823 23426 log.cc:1196] T 19bfb955b9c94fdb8e0381e5b1855248 P e3c6b84864cb4358a9d7e0f2a65ba83d: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-7/wal/wals/19bfb955b9c94fdb8e0381e5b1855248
I20240501 19:58:01.094270 10827 catalog_manager.cc:4830] TS e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): tablet 19bfb955b9c94fdb8e0381e5b1855248 (table rebalance_test_table_2 [id=83036bfcd78e41b484dc84a6bb5ad722]) successfully deleted
W20240501 19:58:01.255335 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.289135 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.291841 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.292564 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.294588 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.295230 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.300691 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.330508 13036 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 5 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20240501 19:58:01.330971 22391 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:58:01.370231 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.381325 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.383282 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.384621 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.391646 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.393051 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.408583 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.414013 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.417464 11113 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 5 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20240501 19:58:01.417846 22391 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:58:01.420953 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:58:01.421399 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:01.433147 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:01.483830 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:01.487553 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.490799 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.493691 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.501305 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.501447 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.512861 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.527287 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:01.531873 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:01.594666 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.609854 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.612164 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.622726 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.623739 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.629087 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.637581 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.666543 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.673713 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.674997 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.678247 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.680429 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.683123 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.683759 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:01.721513 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.757097 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.762173 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.762459 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:01.769647 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:01.774312 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.777087 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.779860 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.790243 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.817225 23421 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:01.817322 23421 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:01.817678 23421 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:01.818454 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:01.818650 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:01.818713 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:01.818830 23421 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:01.865100 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:01.865759 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.869604 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:01.878648 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:01.879145 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:01.888048 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.890244 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:01.910455 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:01.910496 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:01.989845 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:01.990754 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:01.994048 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:01.996026 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.001194 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.006026 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.008277 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.014866 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.026772 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.027371 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.031001 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.031112 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.048354 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.139377 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.141170 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.141810 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.146289 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.146548 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:02.147019 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:02.155207 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:02.201915 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.239882 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.240208 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:02.242844 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.247120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.247498 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.250639 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.261430 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:02.296455 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:02.302248 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:02.312541 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:02.331210 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.366391 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.369153 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.370079 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.372426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.388150 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.388695 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.390362 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:02.390545 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:02.396592 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.401229 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.401584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.402186 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.403648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:02.430606 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.513479 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.513712 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.519271 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.519341 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.528707 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.534274 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:02.554123 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:02.560245 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.610009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.616160 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.617563 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.620755 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:02.624996 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:02.627991 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.629098 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:02.635917 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:02.647491 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 19:58:02.731366 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.735929 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.745453 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.745744 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.750989 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.752318 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.753139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.766695 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.767005 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.767207 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.787636 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.795652 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.885028 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:02.891621 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:02.892647 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:02.895632 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.905575 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:02.915802 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:02.933841 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:02.988834 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:02.993620 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.005417 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.006089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.011449 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.013296 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.093219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.094415 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.102721 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.103574 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.109083 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.124756 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:03.151981 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:03.154021 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.155759 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.161530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.165445 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.175989 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.184513 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.268232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.269304 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.273151 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.282955 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.283694 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.304842 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.380172 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.383850 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.384335 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.386242 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.389042 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.394942 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.468369 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.474421 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.476837 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.479807 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.480190 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.489938 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:03.501335 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:03.540309 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.541894 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.542107 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.542706 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.544108 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.545567 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.639621 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.642675 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.644071 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.645637 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.652617 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.654420 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:03.685184 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:03.701575 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:03.757967 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.759792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.762411 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.777779 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.779590 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.788373 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:03.793767 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:03.844650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.846419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.850569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.855741 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.859073 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.860395 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:03.941531 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:03.941577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:03.943974 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:03.945866 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:03.955498 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:03.959920 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:03.969987 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:03.992837 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.013990 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.049713 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.055877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.056449 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.056867 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.057101 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.073458 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.103919 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:04.117370 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:04.124436 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240501 19:58:04.125057 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.133980 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 19:58:04.134652 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.150971 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240501 19:58:04.151702 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.157948 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 19:58:04.159241 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.162135 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:04.168090 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 19:58:04.168689 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.173163 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.173418 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.182134 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.185079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.230355 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.231087 23421 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:04.231186 23421 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:04.231256 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.231546 23421 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:04.233711 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:04.233851 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:04.233896 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:04.234025 23421 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:04.235438 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.244886 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 19:58:04.245030 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.249640 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.254994 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.262166 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.317432 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.324610 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.326933 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.330520 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.331913 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.333526 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.346307 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.357618 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:58:04.368108 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.395370 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:04.535796 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:04.542389 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.543140 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.545913 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.547225 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.548485 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.554675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.637604 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:04.641711 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:04.643077 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.644557 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.646469 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.647217 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.651479 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:04.658895 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.672713 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.741995 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.743467 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.743697 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.751179 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.751761 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:04.758056 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.760752 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.860641 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:04.864600 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.866477 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.867455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.869213 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:04.874279 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:04.875942 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.886777 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:04.889360 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:04.904354 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.905184 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:04.907495 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:04.907522 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:04.933372 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:04.998543 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:05.000990 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:05.006892 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:05.013060 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.014045 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.016952 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.018494 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.027089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.030404 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:05.035490 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:05.036677 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:05.077754 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:58:05.084429 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:05.108883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.109252 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.110739 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.114393 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.116842 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.135104 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:05.213323 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:05.225761 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.235083 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.237108 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.240535 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.245728 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.248854 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.254810 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.263314 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.279363 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.283675 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.284409 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:05.285544 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:05.296221 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.381505 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.383466 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.390938 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.391497 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.392395 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.392576 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.482635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.485728 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.487381 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.495036 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.495656 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.498781 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:05.508061 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:05.574873 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:05.601908 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.602658 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.603098 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.610877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.612291 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.612550 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.621284 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.636525 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.639926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.644450 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.645562 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.655509 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.741626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.743788 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.749100 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.750578 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.750872 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.751271 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.845504 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.846356 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.848371 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.852322 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.854737 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.856165 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.952816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:05.961585 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:05.963198 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.963917 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:05.967130 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:05.973647 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:05.974014 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:05.987816 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:05.991478 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.006125 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.016407 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.027642 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:06.037353 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.037683 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.201026 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.218886 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.254688 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.310789 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.319149 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.319574 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.320457 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:06.331111 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.342231 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.347110 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.372776 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.375778 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.381824 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.383512 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.383517 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.399020 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.420442 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:06.435848 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.470928 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.471572 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.475155 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.477773 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.482147 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.486439 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.496946 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.565492 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.572332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.572822 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.573321 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.583909 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.592453 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.592945 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.633926 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.644868 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:06.667248 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.672562 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.675776 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.685483 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.685761 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.694381 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.696516 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.735826 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.736557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.738489 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.739929 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.749224 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.758296 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.764667 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.783910 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:06.786053 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:06.829269 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.830945 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.834059 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.834718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.836252 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.854596 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.889221 25197 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:06.889333 25197 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:06.889701 25197 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:06.890606 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:06.890703 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:06.890781 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:06.890915 25197 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:06.924365 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 19:58:06.930881 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:06.934396 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:06.936887 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:06.942209 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:06.942905 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:06.949036 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:06.995216 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:07.008154 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:07.033771 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.038592 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.039180 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.042892 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.050639 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.069303 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.102084 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.103631 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.107194 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.119436 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.120230 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.121079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:07.146672 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.200978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.203243 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.206122 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.206456 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.219178 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:07.224752 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:07.231134 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:07.277734 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:07.294582 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.309193 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.326103 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.327782 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.330402 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.333060 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.337013 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:07.349627 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:07.370517 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.400275 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.404762 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.405752 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.416326 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.418020 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.435915 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:07.442914 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.468775 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.472973 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.473569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.474797 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.475638 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.485533 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:07.488682 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:07.499780 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:58:07.508828 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:07.524617 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.570870 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.571276 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.571359 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.572468 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.572979 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:07.589057 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:07.662443 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.667496 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.669900 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:07.671985 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.673030 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.677930 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.678612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.686908 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.755594 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.760514 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.761307 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.761417 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.762151 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:07.777901 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:07.793828 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:07.925956 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:07.969827 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:07.985399 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:07.987140 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:07.991086 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:07.996038 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.004249 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.010954 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.102994 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.104308 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.107753 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.109339 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.118050 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.121563 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.131074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.135146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.150530 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.153184 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.169517 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.185376 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:08.202837 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:08.260530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.265169 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.265239 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.267249 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.279044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.280418 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.372475 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.378867 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.379366 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.380674 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.381071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.381909 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.473242 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:08.484894 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:08.488217 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.488808 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.490243 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.505509 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.508150 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.512773 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.514876 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.519999 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.530309 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.550220 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.551426 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:08.586207 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:08.616947 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:08.634543 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.636500 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.638881 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.641393 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.642074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.648931 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.733744 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.734798 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.742252 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.748708 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.749614 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.761844 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:08.769819 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:08.780153 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:08.849359 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.850692 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:08.863248 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.864895 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.865939 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.871546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.892875 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.897922 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:08.898339 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:08.908129 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:08.909548 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:08.917876 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:08.933595 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:08.934393 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:08.948179 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.001233 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.003809 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:09.011003 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.016398 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.016919 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.017426 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.019549 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.038579 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:09.076360 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.112792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.117398 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.118146 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:09.118294 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.126076 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.134814 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:09.135735 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.143103 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:09.199033 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 19:58:09.210084 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 19:58:09.241954 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:09.242425 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.244084 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:09.245280 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 19:58:09.245959 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.246780 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.250422 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.261565 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.270927 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.273584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.277016 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.288791 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.288941 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.289263 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.320178 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.322888 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:09.325603 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240501 19:58:09.328936 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.337007 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.352069 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 19:58:09.352823 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 19:58:09.416167 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.418859 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.423986 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.432315 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.438002 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:09.453882 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.454880 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:09.487766 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:09.533370 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:09.608608 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.614761 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.615640 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.616791 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.617588 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.617944 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:09.668280 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.730438 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:09.731743 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.733976 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.734459 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.739970 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.746429 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.750217 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:09.796495 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.811861 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.815867 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.826680 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.826828 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.830435 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.830830 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:09.838912 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.844674 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:09.854238 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:09.896705 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:09.937736 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.942536 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.951934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.961048 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.961995 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.962613 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.972019 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:09.973068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:09.973274 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:09.973872 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:09.994792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.023806 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:10.035302 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:10.035445 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:10.088200 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:10.094696 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.098544 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.099162 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.099844 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.111299 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.130501 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.203439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.204325 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:10.207238 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:10.209445 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.209725 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.222038 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.228338 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.339088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:10.339498 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 19:58:10.342754 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.345180 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.348024 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.348287 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.351603 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.364854 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.377127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.386023 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.392161 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.401520 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.417466 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:10.446671 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:10.469564 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:10.477378 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.478156 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.481644 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.482787 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.489584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.490056 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:10.513798 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:10.576612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.582051 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.582764 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.583148 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.590250 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.590641 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.690440 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.690793 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.692806 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.698211 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.704947 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.707548 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.749020 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.751726 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.752480 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.755057 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.757102 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.766328 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.842149 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.844563 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.849478 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.850163 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:10.850384 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.853847 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:10.880862 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:10.941988 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:10.946219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:10.947714 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:10.948424 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.953279 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:10.974128 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:10.996083 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:11.074767 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.078226 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.082479 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.086483 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.092064 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:11.101657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.104882 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.113629 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.143201 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.143249 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.150370 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.152731 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:11.169590 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.192524 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.253882 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.338018 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.356331 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.370421 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:11.443677 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:11.445302 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.455608 27659 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:11.455718 27659 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:11.456100 27659 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:58:11.456282 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:11.458168 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:11.458398 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:11.458457 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:11.458772 27659 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:58:11.459563 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.466754 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.467340 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.469851 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.469870 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.475405 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:11.476600 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.477793 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.481873 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.497707 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.498726 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.536715 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:11.552448 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.607769 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.611311 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.612533 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.616559 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.618857 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.620536 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:11.627882 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.630610 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:58:11.648067 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:11.707636 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.707806 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:11.709519 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.710139 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.712458 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.717918 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:11.726287 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.732272 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.825855 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.829468 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.836267 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:11.839133 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:11.849015 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.849826 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.850168 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.853329 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.862757 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.872289 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:11.882062 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.883790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.908080 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:11.910394 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:11.930259 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:11.973508 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:11.974790 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:11.976601 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.983700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:11.984488 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:11.994522 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:12.066769 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:12.088029 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.088088 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.091178 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.093057 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.095006 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.114394 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:12.147192 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:12.199904 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.205446 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.205618 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.209803 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.228139 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.228161 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.248399 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:12.249720 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:12.264036 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.264631 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.265444 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.272322 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.277766 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:12.300344 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:12.310058 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:12.353156 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:12.355968 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:12.371500 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.372119 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.373275 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.373277 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.374878 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.382228 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:12.432427 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:12.464772 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:12.474225 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.474907 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.475090 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.476852 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.478655 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.479585 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:12.498245 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:12.567982 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:12.574970 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.575516 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.583209 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.593348 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.597263 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.617672 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:12.625615 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.626292 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.626516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.630616 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.641134 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.669793 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:12.775311 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:12.795634 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:12.860294 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:12.922294 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:12.922930 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:12.929095 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.929487 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:12.930979 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:12.931452 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:12.978708 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:58:13.065536 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:13.224644 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:13.229308 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:13.229792 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.231727 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.237437 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:13.237512 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:13.360796 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:13.446746 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:13.539306 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:13.539821 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:13.540190 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.541316 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:13.541579 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:13.557862 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:13.577489 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:13.712304 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.716118 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:13.717347 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:13.717909 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:13.719852 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:13.720403 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:13.772984 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:13.786250 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:13.819869 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:13.820433 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:13.823206 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:13.827160 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.833060 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.841993 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:13.884202 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:13.893131 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:13.895807 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:13.896054 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:13.902078 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:13.903146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:13.914906 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:13.916126 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:13.930362 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:13.973371 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:13.978621 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:13.982920 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:13.992661 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:14.123929 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.127265 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.129352 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.129850 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:14.133142 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.143704 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.145617 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.146927 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.236768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.245071 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:14.245910 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:58:14.246798 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.247599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.251490 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.253768 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.266120 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:14.275696 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 19:58:14.277419 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.289956 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.300294 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.305006 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.308209 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.312005 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:14.325276 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:14.358487 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 19:58:14.359174 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 19:58:14.388406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.390369 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.392491 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.394538 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.397889 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.403115 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:14.415308 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 19:58:14.482321 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:14.490643 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:58:14.491552 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.497673 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.499224 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.499980 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.501058 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.501294 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:14.519258 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:14.548249 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:14.577275 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:14.594563 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:14.602180 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.602228 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.602726 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.605957 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:14.609092 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.614367 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.625716 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.654093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.658103 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.664626 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.668061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.671854 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.675770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:14.699491 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:14.719085 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.750969 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:14.751165 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:14.751261 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.755352 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.758661 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:14.761133 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.761981 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.762081 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.769901 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:14.779424 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.854259 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:14.858680 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:14.861964 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:14.870047 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.870674 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:14.885891 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.972653 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:14.979736 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:14.980221 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.986342 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:14.998282 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.003127 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.007411 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.019658 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.035660 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.043097 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.043751 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.049976 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.050244 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:15.079269 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:15.130234 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.132508 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.135320 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.141779 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.142874 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.164825 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:15.179960 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:15.196882 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:15.242921 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.244577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.248777 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.249094 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.253773 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.260819 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:15.320811 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:15.364202 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:15.373776 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.379051 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.379351 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.383811 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.390134 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.398223 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.399190 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.404871 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.419423 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.424983 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.425199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.428731 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.515903 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.516146 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.518430 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.519073 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.519433 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.534971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:15.598269 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:58:15.603363 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:15.713696 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.714262 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.719178 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.721060 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.737224 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.743199 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:15.818025 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:15.822588 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.824326 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.825181 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.826548 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.843153 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:15.843865 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:15.878696 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:15.922613 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:15.928103 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:15.936045 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.936484 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:15.936517 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:15.945153 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:15.954056 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.040561 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.045814 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.048002 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.049453 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.049582 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.051890 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.102672 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:16.103893 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:16.104033 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.107347 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.117893 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.121773 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.136440 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:16.208228 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:16.210979 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:16.212379 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.218533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.218889 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.220714 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.229494 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.234644 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:16.319239 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:16.323860 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.325717 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.333043 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.333302 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.335431 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.337273 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.339835 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.407663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.408012 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.410140 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.410387 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.414644 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:16.429092 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.442431 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:16.447118 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.478878 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.481225 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.484279 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.484272 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.492741 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.501837 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:16.521893 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:16.567416 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.583261 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.583889 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.587869 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.588400 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.589067 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.603351 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:16.668188 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:16.691128 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:16.691614 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.695238 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.697511 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.701040 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:16.705278 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 19:58:16.709375 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.709796 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.775898 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.783655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.784957 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.786515 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.792564 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.803432 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:16.820593 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:16.854365 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.854838 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.856139 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.859977 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.860512 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.869318 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:16.884636 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:16.893049 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:16.959374 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:16.963289 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:16.963351 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:16.965058 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:16.965735 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:16.981750 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.057749 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.060631 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.067354 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.072209 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.072850 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.078703 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.083015 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:17.083317 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:17.083882 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:17.142704 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.143703 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.144263 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.145934 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.146721 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.152940 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.162460 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.174266 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.198227 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.203903 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.236445 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.239641 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.243088 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.243489 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.254645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.256544 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.303575 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.390977 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.428711 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:17.452469 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.452667 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.462426 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.465016 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.465703 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.482441 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.488858 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.503453 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:17.554888 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.555752 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.559784 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.570647 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.570900 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.583292 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.655879 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.656963 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.657881 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.658090 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.659722 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.666397 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.772624 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.774659 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.783102 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.783545 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.785703 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.785781 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.802809 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.815160 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:17.823016 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.824730 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.829365 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.829944 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.841818 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.842581 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:17.878057 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:17.919034 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:17.920555 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:17.926784 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:17.929085 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:17.931964 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:17.947319 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:17.985965 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:17.986694 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:18.022182 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.025461 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.026013 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.027909 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.033053 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.047760 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.141690 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.146775 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.161767 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.166800 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.167200 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.179263 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.179507 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.185624 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.186657 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.199612 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.206456 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.231963 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.252069 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:18.313335 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.314644 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.321077 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.323685 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.327805 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:18.330299 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:18.335570 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.349757 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:58:18.420382 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:18.421841 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.424538 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.427253 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.431578 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.434461 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.436115 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.440001 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:18.537218 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.537312 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.537528 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.538796 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.540761 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.547888 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.558310 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:18.562220 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.573536 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.576241 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.578809 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.598917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.606017 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.607542 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:18.626821 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:18.667507 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:18.685880 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:18.687455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.688100 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.690678 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.692011 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.707000 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.719851 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.801478 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.804096 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.805925 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.807092 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.808868 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.833070 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.903537 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.906538 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.909281 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.917382 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.921197 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.929666 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:18.963222 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:18.965020 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:18.970516 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:18.971320 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:18.972347 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:18.990101 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:18.993677 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:19.020843 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.021133 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:19.142262 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.180096 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 19:58:19.221869 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.222121 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:19.223819 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.226507 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.228997 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.229662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:19.253005 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.318749 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:19.323697 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.326578 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.333410 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.335243 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.347873 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:19.357095 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:58:19.360230 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.384538 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 19:58:19.401908 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:19.423321 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.425772 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.426628 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:19.427661 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.428529 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.433215 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:19.441468 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.441761 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:19.496745 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:58:19.498214 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 19:58:19.534266 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.536857 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:19.537659 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 19:58:19.544689 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.545342 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.547710 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:19.557276 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.557652 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.567329 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.579793 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:19.580674 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:19.584101 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.585108 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:19.602011 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.604393 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.612473 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:19.621294 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.638897 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.638906 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 19:58:19.681397 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:19.685050 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.685878 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.687661 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.689630 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.698493 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:19.698546 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.707857 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.790547 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.790743 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.793285 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:19.795943 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.798048 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.805431 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.808343 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.912629 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.916306 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.919924 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.928488 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.939790 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:19.942457 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:19.943194 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:19.945991 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:19.947423 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:19.953231 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:19.953501 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:19.969604 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:19.985431 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:19.987318 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.068992 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.072713 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.073324 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.075233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.075304 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.083786 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:20.093557 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:20.179416 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.180898 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.182937 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.186290 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.194787 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.195199 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:20.212417 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:20.295140 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.299221 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.301478 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.302737 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.305140 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.315655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.321298 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.327613 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:20.337869 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:20.341259 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.345992 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.346669 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.356009 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.444959 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.445042 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.447383 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.449779 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.464203 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.472656 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:20.525427 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:20.545893 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.547187 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.547470 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.554096 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.561494 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.570156 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:20.577950 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:20.585120 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:20.668630 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.671180 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.671931 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:20.675160 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.678090 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:20.682770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:20.684303 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.689046 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:20.713953 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:20.714102 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 19:58:20.716070 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.724421 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:20.724879 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:20.729635 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:20.757705 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:20.930665 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:20.963712 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:20.985921 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:21.000715 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:21.038324 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.041808 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.049196 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.050060 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.050393 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.050537 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.052321 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.052522 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.053429 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.077399 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.077623 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.085945 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:21.153533 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:21.155202 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.182080 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.187875 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.189465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.193255 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.195469 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.210568 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:21.270361 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.297551 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.300319 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.301692 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.302037 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.310881 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.317888 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.402755 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.405367 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.410267 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.413398 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.418232 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.448014 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.449460 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:21.450888 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.457319 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.458264 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.461503 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.466662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.470062 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:21.503021 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:58:21.504438 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:21.542184 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:21.546795 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:21.562831 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.564314 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.569088 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.569594 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.595207 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.597191 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:21.623292 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.673219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.676218 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.676636 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.679692 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.681212 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.703243 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:21.741017 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.770623 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.776410 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.777189 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.778940 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.785473 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.793947 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:21.824833 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.837720 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.838702 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.845320 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:21.847263 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.855738 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.859069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.940433 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:21.946723 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:21.946837 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:21.949093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:21.950341 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:21.954478 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:21.957332 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:21.958911 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:21.964180 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:21.981801 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:21.993957 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:22.000128 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:22.011051 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:22.046660 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:22.048219 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:22.064886 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.067672 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.069106 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.072330 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.073246 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.073501 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.131228 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.135046 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.136183 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.143289 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.149847 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.151017 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.225122 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.226941 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.231055 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.238214 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.251027 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.253734 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:22.302444 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:22.333276 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.335259 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.335948 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.339303 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.349071 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.355213 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:22.420645 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:22.435369 30285 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:22.435489 30285 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:22.435863 30285 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:22.437788 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:22.439052 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:22.439424 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:22.439482 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:22.439702 30285 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:22.445531 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.446841 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.457370 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.460536 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.462517 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.477587 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.486490 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.495172 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.500288 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.509492 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.514564 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.523766 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:22.536087 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:22.743517 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:22.816550 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.817543 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.820139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.838963 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.840339 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.848791 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.849879 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.850061 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.866357 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.872334 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.873317 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:22.874480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:22.934213 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:22.936292 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:22.957906 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:22.960513 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.962822 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:22.963201 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:22.974701 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:22.978299 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:23.007289 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:58:23.058914 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:23.073371 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.074263 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.074837 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.084024 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.085337 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.087040 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.194310 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.200522 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.201007 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.201371 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.209360 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.216261 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.219699 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.224159 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.239599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.257180 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.259828 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:23.279176 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:23.280684 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.350426 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.351667 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.353440 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.358589 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.359694 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.364769 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:23.389428 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:23.391921 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:23.415714 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:23.431056 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:23.450982 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:23.454371 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.454457 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.455649 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.455969 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.457512 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.465968 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:23.504519 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:23.534075 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:23.590482 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.590754 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.597693 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.602020 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.602703 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.610123 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.629678 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.637406 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.637997 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.645071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:23.645504 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:23.651237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.658053 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.741685 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.743647 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.753762 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.754952 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.765152 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.767513 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.851132 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.851173 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:23.857569 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.858567 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.860247 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.861059 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:23.883401 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:23.901192 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:23.951802 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:23.961567 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:23.963565 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:23.964965 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:23.970974 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:23.978227 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:23.999785 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.000283 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.019012 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.028628 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.030889 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.038812 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.040897 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.055114 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:24.103160 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:24.132489 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.133685 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.138938 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.150607 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.154093 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.161087 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.176394 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 19:58:24.240729 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.241389 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.244453 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.250012 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:24.254313 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.262352 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.277472 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.286156 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:24.343690 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.345194 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.351051 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.352713 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.356272 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.360594 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.375416 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 19:58:24.376169 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:24.399003 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:24.413616 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:24.416369 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:58:24.457778 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:24.506305 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 19:58:24.514021 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:24.524303 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:24.551664 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 19:58:24.559759 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:24.562765 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.563103 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.564235 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.573849 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.578534 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.590790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.642354 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:24.667791 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.674418 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.675969 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:24.678349 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.682428 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.682942 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.684808 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.697440 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:24.699038 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:58:24.704521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.709064 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.717095 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.734760 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.734978 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.760459 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:24.825001 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.827404 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:24.830394 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.831687 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.840512 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.842643 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:24.901304 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:24.933323 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:24.935954 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:24.937888 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.938011 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:24.939978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:24.949208 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:24.964607 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 19:58:24.994231 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:25.025010 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:25.045570 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.046814 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.055045 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.056231 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.057308 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.062849 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.095635 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.100363 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.100378 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.115155 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.120926 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:25.121336 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:25.123404 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:25.194607 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:25.206190 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.206866 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.209625 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.211773 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.213083 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.213402 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.308698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.310985 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.311599 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.313675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.323158 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.328966 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:25.348511 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:25.413945 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.416164 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.423944 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.424824 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.447901 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.467069 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.475015 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.479177 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.482472 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.488893 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:25.490650 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:25.498977 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.505296 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.591063 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.596094 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.597088 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.600921 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.604257 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.617830 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:25.644888 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:25.695245 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.696570 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.701776 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.702108 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.708933 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.714540 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:25.736729 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:25.810426 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.812357 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.815732 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:25.820240 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:25.828841 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.831035 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.833278 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.847098 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:25.861023 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.864290 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.873980 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:25.875288 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.887485 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:25.919163 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:58:25.946965 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:25.965669 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:25.965902 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.970618 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:25.970783 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:25.971802 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:25.977906 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:25.997028 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:25.999720 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.004225 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.090276 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:26.091401 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.145967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.149552 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.153712 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.161049 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.162925 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.169366 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.178555 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.193832 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.285846 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.287235 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.287420 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.289721 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.301376 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.319442 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:26.329731 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.390936 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.393745 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.395229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.398528 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.401288 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.405414 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.410513 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:26.471940 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 19:58:26.485599 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.489143 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.492697 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.494874 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.497202 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:26.498983 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.530968 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.543813 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.544704 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.548662 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.550662 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.561262 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.565271 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.615411 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.655789 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.656934 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.661386 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.661638 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.670478 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.684000 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:26.704285 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.756233 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.761104 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.762498 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.767519 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.769510 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.769686 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.782377 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.816864 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.826550 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.873060 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.881290 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:26.882153 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.882648 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.890362 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:26.901449 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.901994 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:26.903085 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.904011 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.924971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:26.928963 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:26.929512 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:58:26.929915 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:26.934038 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:26.939410 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:26.939517 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:26.941394 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:26.991044 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:27.041200 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.041649 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.044679 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.053191 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.055013 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.066044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.139552 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:27.150038 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.150472 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.152503 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.157001 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.162978 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:27.169129 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.178591 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.228650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.230383 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.232702 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.232864 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.233424 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.257103 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.305917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.309435 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.314671 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.314960 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.316422 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.330672 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.401736 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:27.412775 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.414201 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.417820 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.430178 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.430953 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.432739 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.450049 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:27.518239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.519361 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.523169 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.527170 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.529645 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.538756 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.552338 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:27.595465 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:27.597750 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.598995 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.609886 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.611006 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.614213 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.619920 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.620591 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:27.668828 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.672932 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.675103 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.680297 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:27.681569 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.682807 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:27.840720 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:27.907610 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:27.909493 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.915087 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:27.915546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:27.916007 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:27.917052 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:27.950402 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:28.004750 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.007826 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.016162 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.016391 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.021632 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.046072 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:28.058638 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:28.122033 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.122504 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.123399 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.124461 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:28.127075 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:28.128396 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.136797 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:28.208220 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:28.239925 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.241084 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.245462 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.252128 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.254273 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.269110 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.279605 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.280927 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.283792 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.298089 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.300940 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:28.324601 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:28.325204 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.401307 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.402699 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.406533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.407792 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.408166 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.411892 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:28.422047 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:28.454824 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:28.469959 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:28.481899 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:28.497292 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.498234 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:28.498914 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:28.500021 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:28.505174 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:28.507387 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.508123 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:28.514221 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:28.516324 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:28.535180 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:28.621258 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.633294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.633333 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.633549 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.635766 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.636860 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.641722 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.649103 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.658797 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.660805 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.681476 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.686005 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.766362 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.769393 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.772437 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.774284 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.775066 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.776428 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:28.864706 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:28.875502 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:28.877182 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:28.879051 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.879737 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:28.880290 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.889420 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:28.977696 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:28.989961 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.990341 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:28.990638 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:28.993347 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.007903 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.008280 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.013936 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:29.017222 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 19:58:29.017621 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.018389 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.032127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.055613 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.061213 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.117678 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.140743 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.142232 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.142345 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:29.144337 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.145574 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.153820 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.169406 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.210428 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:29.215284 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.245820 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.246826 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.248684 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.250430 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.254428 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.262992 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:29.326269 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.368172 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.369331 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:29.369761 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.370887 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.373878 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.377055 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.377966 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.388842 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:58:29.389544 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.391491 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.399300 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.402061 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.408831 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.420421 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.421737 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:29.428898 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:29.482230 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 19:58:29.499577 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:29.523103 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:29.528739 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:58:29.613793 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 19:58:29.614413 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:29.626042 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:29.663740 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.726902 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.729452 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.733839 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.735692 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.737010 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.746368 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.753121 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.757166 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.768885 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.769002 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.776684 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 19:58:29.779300 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.794178 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:29.818598 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.874279 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:29.874545 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.875056 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.875870 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.879041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:29.879343 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:29.974320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:29.976903 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:29.977596 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:29.980389 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:29.982851 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:29.986963 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:29.988366 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:30.002538 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 19:58:30.011993 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:30.028209 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:30.034951 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:30.089044 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.091229 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.094379 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.096393 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.103231 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.115233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:30.138290 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:30.158769 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.159214 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.160921 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.161803 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.169453 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.180133 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.255709 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.258339 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.261426 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.267975 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.269301 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.275003 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:30.360716 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:30.364133 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.369553 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.370679 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.371064 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.379642 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.396306 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:30.448661 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:30.452262 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.453214 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.461321 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.468442 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.470738 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.480639 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.515553 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.521728 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.522182 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.524499 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.526007 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.537396 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:30.594882 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:30.619537 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.620734 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.622318 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.624235 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.629427 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:30.633940 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:30.645424 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:30.653569 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:30.671106 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:30.715516 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 19:58:30.726572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.730868 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.734282 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.737388 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.764345 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.765954 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:30.808452 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:30.814524 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:30.816195 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.818172 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.819084 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.820075 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.822320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:30.846014 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:30.892755 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.895483 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.895694 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.895936 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:30.909708 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:30.913812 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:30.988128 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:30.988749 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:30.990355 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:30.999322 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.000758 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.014022 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.015866 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:31.026006 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.051098 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.086355 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:31.110567 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.123755 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.126463 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.135227 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.155184 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.156459 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.165406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.165423 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.172165 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.173337 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.181964 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.201575 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:31.216246 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:31.220276 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.348481 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:58:31.382094 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:31.463622 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.465572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.467308 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.470719 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.473909 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.485790 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.537724 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.541453 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.542009 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.546826 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:31.555469 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:31.559912 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.562219 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.620162 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.624210 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.627087 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.627633 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.628582 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:31.639914 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:31.640923 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:31.644230 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.691637 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.704813 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:31.727064 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.729321 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.732563 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.733650 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.734315 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.744225 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:31.752179 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.758160 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.777554 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:31.809197 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:31.833729 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.834584 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.835119 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.838097 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.845037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.845285 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.899415 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.902803 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:31.910379 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:31.912561 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.926223 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.929368 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:31.971356 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:31.971483 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:31.971645 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:31.982108 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:31.983139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.000520 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.000521 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.084764 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.088935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.091637 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.099906 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.100333 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.109467 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:32.160804 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:32.167970 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:32.176331 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:32.212574 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.213172 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.213737 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.215911 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.216018 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.237906 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:32.263295 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:32.275285 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.280716 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.281158 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.283051 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.295822 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.310395 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:32.325156 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:32.370201 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.372943 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.377976 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.378350 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.382381 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.390326 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.471141 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.475394 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.477505 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.480471 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.486426 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.498363 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:32.539680 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:32.554762 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:32.590878 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.592007 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.594965 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:32.606191 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:32.613996 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.622346 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.625238 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.628373 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:32.633508 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:32.638216 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:32.646054 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.658213 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.660931 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.662418 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.671722 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:32.675772 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:32.716149 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:32.740556 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.743618 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.746654 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.749310 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.751645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.753000 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.844805 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.845085 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.845729 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.855343 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.856508 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.868340 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:32.947870 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:32.953367 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:32.953869 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:32.957803 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:32.965189 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:32.969182 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:32.975003 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:33.127395 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:33.149971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.154578 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.156495 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:33.157763 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:33.163823 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.165179 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.167210 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:33.177382 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:33.242560 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 19:58:33.258463 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.259645 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.260389 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.261031 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:33.262338 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:33.276988 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.281894 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:33.306295 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:33.329664 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.334369 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.335547 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.340444 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.346311 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.352546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.408674 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.410740 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.411523 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.414193 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.414553 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.415105 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:33.477267 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:33.508123 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.509605 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.509925 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.511019 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.518362 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.525313 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:33.561354 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:33.570628 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:33.614619 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.616299 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.619168 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.623382 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.636193 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.651006 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:33.665095 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:58:33.683279 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:33.702711 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.703419 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.712953 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.721843 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.724110 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.724526 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:33.725620 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:33.779232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.786085 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.787143 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:33.789618 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:33.798696 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:33.803596 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.809470 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.891444 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:33.891538 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:33.891631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:33.903986 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.904250 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:33.904482 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:33.970690 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:33.999122 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.002688 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.009977 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.009999 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.025167 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.029479 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.085939 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.092711 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.095798 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.096433 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.099299 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.106541 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.107050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:34.133957 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.184768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.187994 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.188759 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.192114 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.197103 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.197500 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:34.245306 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:34.281652 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.292462 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.294922 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:34.295117 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.295672 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.307565 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.307833 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.311417 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:34.350286 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:34.400430 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 19:58:34.401022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:34.401391 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:34.404644 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.415953 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.417213 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.421088 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.421721 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.429550 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.462076 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.467016 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.468199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.471638 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.474900 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.481376 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.557818 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.562229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.564074 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:34.565549 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:58:34.566814 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.577819 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.579061 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.610905 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:34.634060 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:58:34.650401 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:34.652148 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 19:58:34.774297 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.776026 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.776721 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.779528 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.779676 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:34.781069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.796233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.829605 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:34.836122 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:34.868203 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.873283 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.873977 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.875412 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.876470 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:34.878311 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.882119 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:34.908002 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:34.967834 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:34.969755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:34.972996 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:34.974702 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.974844 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:34.981763 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:35.008553 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:35.013594 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 19:58:35.014150 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:35.037142 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:35.077627 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 19:58:35.104555 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.105273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.111737 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.114014 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.115092 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.121888 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.125810 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.128362 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.133579 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.153486 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.155171 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.174351 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:35.183586 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:35.260473 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.262004 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.263084 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.264679 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.273025 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.273389 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:35.358886 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:35.367674 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.369261 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.370826 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.373152 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.381932 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.382339 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.477888 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:35.479424 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:35.481508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.486624 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.490633 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.496937 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.499923 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.503012 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:35.505218 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:35.505846 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.506999 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.521023 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.522640 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.535538 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:35.598949 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:35.606225 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:35.612516 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 19:58:35.629947 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.636808 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.638856 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.639209 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.639827 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.647488 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.734113 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.735594 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.736339 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.736716 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.739362 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.743103 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.851858 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.856330 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.859004 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.862179 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:35.865003 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.874809 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:35.885250 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.885372 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:35.893541 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:35.894181 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.904390 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:35.911314 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:35.980149 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:36.010293 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.013876 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.015273 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.018297 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.026243 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:36.036484 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.036823 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.048203 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:36.062552 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:36.114145 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.117069 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.120592 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.122226 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.133881 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.134433 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.152313 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:36.178418 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.225353 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.229012 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.233108 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.235476 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.242700 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.244040 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.255395 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.288240 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.290273 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.290638 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.294456 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.304704 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.306063 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:36.325244 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:58:36.437348 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:36.472604 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:36.545533 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.546327 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.550488 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.554486 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.555569 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.567489 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.578657 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.585191 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:36.587839 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.628868 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.630203 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.633193 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.633981 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.640413 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:36.642503 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.650189 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.726210 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.726660 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.727540 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.727897 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.736125 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:36.736289 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:36.754102 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.838238 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.839187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.843657 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.851596 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:36.857153 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.858342 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.916942 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:36.918543 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:36.923713 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.927775 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:36.936491 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:36.948016 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:36.980475 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:36.983537 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.002480 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.009642 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.009959 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:37.013044 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.013484 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.021426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:37.033779 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:37.033895 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:37.045694 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:37.085680 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.107456 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.111649 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.113351 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.113828 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.124822 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.131493 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:37.203944 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.227923 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.234452 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.239387 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.246866 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.252975 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.262068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.292517 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:37.293561 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.295037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.297151 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.305101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.305652 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.309062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:37.338022 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:37.372133 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.389883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.391260 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.395197 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.396520 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.409725 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.411198 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:37.414590 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.467921 5094 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:37.468029 5094 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:37.468423 5094 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:37.469559 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:37.470710 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:37.470775 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:58:37.471295 5094 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:37.492799 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.496309 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.499492 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.506093 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.507701 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.520349 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:37.521602 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:37.609412 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:37.615077 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.620385 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.632265 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.632512 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:37.634280 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.636225 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.639933 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.641381 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.643317 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.656746 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.669430 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.678607 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.686874 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.766373 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.770388 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.770388 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.771855 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.777096 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.777855 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:37.796433 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:58:37.812232 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:37.871776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:37.875125 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.875362 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:37.876386 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.882908 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:37.886082 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:37.888801 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:37.971024 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:37.982805 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:37.986274 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:37.987821 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:37.995023 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.005637 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.023496 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.033262 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.042572 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.044122 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.045490 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.054451 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.073390 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:38.106647 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:38.120586 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:38.145237 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:38.344156 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.344770 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.351332 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.351595 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.356487 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.378718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.390972 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.402323 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.404019 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.404332 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.406982 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.434957 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.507942 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.509714 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.510609 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.511518 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.515010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.521049 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:38.536037 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:38.565714 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:38.611790 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.614058 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.615113 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.623135 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.623667 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.625614 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:38.632601 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:38.636008 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:38.667101 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:38.699770 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:38.712563 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 19:58:38.718489 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.724841 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.725893 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.732962 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.740964 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.766355 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.774618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.775408 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.776561 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.778626 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.781638 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.791390 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:38.813112 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:38.831691 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:38.880955 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.882900 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.885450 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:38.889106 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:38.889441 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:38.899744 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:38.900375 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:38.958505 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:38.988948 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:38.993782 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:38.994345 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:38.997277 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.009081 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.010229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:39.059451 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:39.064107 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.106127 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.112452 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.115325 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.122139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.126422 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.151463 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.160545 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:39.160573 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.161804 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.166483 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.173326 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.174010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.182389 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:39.203450 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:39.268020 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.268573 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.283324 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.286125 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.289846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.293926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.379887 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.381796 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.384070 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.384987 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.385582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.404170 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:39.427034 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:39.439646 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:39.482616 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 19:58:39.487497 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.487917 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.488138 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:39.497608 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.500901 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.502606 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.513357 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.524204 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:39.526604 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:39.527356 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:39.533823 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.536247 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:39.537465 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.539651 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:39.543184 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.557960 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.559230 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:39.598461 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:39.614166 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:58:39.636368 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.637809 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.643472 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.645763 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.646992 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.647959 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:39.656596 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 19:58:39.692637 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 19:58:39.736635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:39.737486 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:39.738809 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.742780 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.746124 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:39.751073 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.756258 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.844352 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:39.851938 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.853046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:39.854955 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:39.855046 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:39.865625 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:39.891695 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:39.891997 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:40.030771 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:40.042937 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.045867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.049098 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.050259 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.051529 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.061391 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.130322 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:40.149453 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:40.155970 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.159900 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.161377 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.162446 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.165380 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.176330 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.200889 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:40.208540 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.209726 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.209942 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.214810 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.226819 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.237804 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:40.250020 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:58:40.303778 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 19:58:40.307009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.307070 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.309994 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.317225 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.317294 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.317706 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.357525 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:40.358287 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:40.417870 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.418663 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.421278 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.423851 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.431044 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:40.431481 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.442788 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.481832 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:40.526607 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.532125 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.533200 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.541637 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:40.542891 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:40.542937 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.542979 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.574267 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.587193 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.596289 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.597757 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.603261 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.607576 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.676366 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.678853 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.679975 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.680418 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.682626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.697852 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.777594 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:40.778453 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:40.783200 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.790966 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.793984 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.795454 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.796182 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:40.870836 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:40.902778 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.908859 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.909648 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.919553 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.924211 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.932307 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.932595 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:40.943465 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:40.953155 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:40.955374 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:40.964099 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:40.972478 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:40.985596 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:41.049073 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 19:58:41.064245 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.064268 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.071297 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.071753 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.073256 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.074299 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:41.148100 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:41.165442 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.169342 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.170310 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:41.178045 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.180265 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.182101 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:41.211280 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.288352 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.290357 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.292973 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.297250 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.311800 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:41.312074 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.316252 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.321453 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.322939 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:41.335103 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.342936 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.349260 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:41.393101 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:41.403316 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:41.438817 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.440367 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.441805 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:41.442716 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:41.447399 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.448056 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.453066 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:41.454496 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.457572 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:41.480836 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.636560 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.639827 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.640534 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.640826 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.646325 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.655093 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:41.664364 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:41.732976 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:41.747270 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.751120 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.753543 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.763101 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.765105 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:41.767799 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.769996 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:41.801528 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.854434 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.860435 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:41.862382 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.862967 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:41.863236 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.864775 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:41.879007 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:41.939563 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:41.964957 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:41.970094 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:41.974460 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:41.978461 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.007050 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.007627 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.016223 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:42.026852 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.032301 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.035318 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.036084 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.040177 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.061740 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:42.100741 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:42.130052 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.142560 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.145736 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.146553 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.153136 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:42.157284 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.172602 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.174561 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:42.182873 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:42.233748 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.247977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.249393 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.258033 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.259768 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.260149 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.276870 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:42.290766 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:42.303478 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:42.364171 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.371824 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.372462 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.385946 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.395928 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.409232 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.415802 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.416538 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.418159 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.420320 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.420749 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:42.434988 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:42.438861 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.532263 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.532733 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.533012 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.533259 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.533674 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.546322 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.633268 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.634804 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.639690 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.641871 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.643190 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.653371 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:42.655182 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:58:42.663791 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.743247 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.747548 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:42.751278 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:42.755945 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.756215 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:42.757252 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.760264 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.763492 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.770267 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.774544 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.797612 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:42.799525 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:42.801816 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.804116 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.804862 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.904896 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:42.905287 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:42.905674 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:42.908794 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:42.911569 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:42.913169 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:42.992009 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:43.008504 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.008926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.016285 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.020836 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.022039 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.022123 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:43.056558 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:43.117635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.124779 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.125674 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:43.136633 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:43.149967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.149967 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.151700 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.154994 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.157101 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.162178 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.183915 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.192140 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.196252 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:43.274857 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:43.365892 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:43.489228 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.490958 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.493386 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.499266 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.511947 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.517719 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.524235 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.544497 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.545691 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.548451 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.548990 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.551079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:43.568305 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 19:58:43.650038 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.650291 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.655366 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.657436 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.661052 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.663548 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:43.664330 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:43.716003 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:43.757517 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.759212 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.761961 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.766808 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.776067 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:43.777678 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:43.798462 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:43.807839 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:43.826715 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:43.856686 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.860952 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.861552 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:43.864481 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:43.870239 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:43.872431 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.879729 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.881072 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.937273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:43.940021 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:43.942528 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:43.950871 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:43.951913 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:43.956210 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:43.999887 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.040992 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.041111 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.041420 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.045388 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.048542 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:44.057358 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.057457 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.057633 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.157747 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.166697 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.172304 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.173738 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.173801 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.176342 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.200173 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.249130 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.255795 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.259335 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.261065 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.263938 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.267822 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.278964 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:44.332372 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.337312 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.338218 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.342599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:44.346242 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:44.346776 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.348969 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.356225 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.436339 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.440569 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.440971 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.450634 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.451002 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.451438 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.498848 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:44.518226 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.548079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.548188 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.548717 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.551963 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.564159 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.571179 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:58:44.572160 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.577533 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.607981 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.613866 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.617497 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.619233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.626490 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:44.628057 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:58:44.632321 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.656849 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 19:58:44.661453 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.704612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.706032 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.707517 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.710175 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:44.713745 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.727092 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.729104 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:44.787520 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:44.804278 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.810436 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.814646 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.817852 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.821853 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.830448 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:44.838440 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:44.843222 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 19:58:44.865674 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:44.873919 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:44.886873 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:44.925880 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.926609 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.930677 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.939548 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.952205 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:44.958729 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:44.966468 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.973505 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:44.982264 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:44.988828 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:44.991993 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.020354 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:45.021152 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:45.135798 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:45.267446 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:45.268705 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:45.273996 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.274320 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.278452 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.279489 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.287256 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.289891 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:45.303369 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:45.354373 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.357686 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.358326 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.359321 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.365808 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.367233 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.435356 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.438453 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.439594 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.442806 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.443450 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.445101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:45.480010 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:45.483229 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 19:58:45.497272 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:45.537271 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.543099 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.543196 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.545903 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.546710 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.551764 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:45.554332 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:45.610493 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:45.652635 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.655606 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.657024 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.657528 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.661537 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:45.667245 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:45.685725 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 19:58:45.687884 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.707423 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.708956 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.715728 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.719355 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.724141 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.745781 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.795245 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.796778 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.801293 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.811439 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:45.813077 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.813722 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.903769 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:45.908797 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.909394 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:45.909926 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:45.912457 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:45.913089 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:45.940086 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:58:45.984462 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:46.026283 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.029364 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.029477 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.030715 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.042565 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.043237 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.087630 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.089788 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.093771 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.099193 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.102803 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.119482 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.141515 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:46.178264 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.183333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.185590 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.185811 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.190068 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.194644 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.207921 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.274142 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.289866 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.294688 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.297598 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.303362 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.304636 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.305061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.418123 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.422487 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.429880 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.431488 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.434307 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.437793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.440992 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.444344 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.449672 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.461634 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.473773 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:46.474694 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.483381 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:46.486210 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.486356 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:46.492154 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.557440 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.558669 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.560168 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.565047 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.577240 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.582325 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.591333 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.596258 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:46.616806 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:46.629614 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.667099 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.668274 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.671233 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.678251 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.688516 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.690107 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.741829 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.770440 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.772877 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.774597 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.776113 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:46.781167 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.784632 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:46.807560 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:46.848587 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:46.972101 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:46.974004 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:46.977883 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:46.982831 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:46.991863 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.004590 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:47.031826 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:47.092818 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.100024 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.102818 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.104036 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.117950 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.120105 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.121160 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:47.133949 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:47.134996 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.135783 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.150861 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.161434 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.174450 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:47.175393 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:47.224647 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:47.237511 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:47.254523 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.256420 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.257258 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.257306 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.258723 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.269975 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:47.310231 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:47.311807 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:47.366269 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.367626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.370131 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.370716 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.373260 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.375016 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:47.412051 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:47.474270 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.480701 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.484503 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.484625 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.489522 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.504336 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.504805 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.518662 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.522516 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.528323 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.530242 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.533389 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:47.545378 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:47.545449 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:58:47.545567 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:47.553040 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:47.627326 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.628293 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.629156 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.632968 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.633080 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.650141 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:47.710014 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:47.729228 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.730859 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.734381 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.744693 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.744804 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.748139 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.847887 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.851035 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.853323 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.864425 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:47.864820 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:47.874331 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:47.874532 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:47.876976 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.880864 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:47.885672 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:47.890914 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:47.915359 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:47.917693 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:47.921259 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.011756 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.016593 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.020192 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.020293 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:48.021764 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.029855 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.032714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.084468 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:48.119087 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.121589 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.122344 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.124205 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.130589 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:48.144852 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.158164 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.192824 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:48.217253 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.218703 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.218730 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:48.218837 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:48.223739 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.227017 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.246793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.287096 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.288868 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.291641 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.292253 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:48.302268 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 19:58:48.305855 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.311128 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.387158 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.398738 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.398876 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.399632 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.404207 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.405517 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.414476 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.528997 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.529603 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.530144 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.533768 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.537832 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.546030 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:48.570215 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:48.711117 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.711215 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.714035 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.715610 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:48.717548 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.724334 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.737152 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:48.738533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.825093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.829056 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.833541 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:48.833557 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.833843 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.836095 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:48.857690 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.860965 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.891515 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.896715 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.896942 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.898846 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.909929 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.925071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:48.954634 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:48.958422 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:48.980556 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:48.984256 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:48.984732 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:48.989114 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:48.990371 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.021587 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.100072 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.100533 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.101369 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.104357 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.110874 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.113965 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:49.119874 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.148310 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:49.179150 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.200500 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:49.214011 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.216012 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.219358 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.220189 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.226516 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.242197 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.252116 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.255200 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.255200 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.256776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.268570 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:49.275542 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:49.293957 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:49.351385 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:49.372237 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.375247 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.380455 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.381181 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.393229 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.393744 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:49.464043 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:49.474512 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.474737 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.478731 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.479328 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.481496 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.483276 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:49.494871 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.563918 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.565696 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:49.583470 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.589916 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.595275 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.595965 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.600875 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.608603 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.655552 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.656291 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.657342 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.657582 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.659564 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.675033 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:49.682818 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:49.702857 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 19:58:49.703495 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:49.735179 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:49.735321 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.737977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.744359 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.744561 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.744935 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:49.794149 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:58:49.794761 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:49.820158 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:49.836587 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.839540 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.839818 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.850189 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.858662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.860069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:49.867363 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.905783 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:49.920167 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 19:58:49.934840 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:49.951292 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:49.951987 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:49.953110 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:49.953650 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.964375 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:49.966308 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.024087 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.031893 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.036650 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.038873 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.039904 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:50.045343 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:50.049556 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:50.095256 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:50.113929 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.117254 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.118692 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.126107 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.126850 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.129874 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:50.194316 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:58:50.257776 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:50.345050 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.347404 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.349846 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.351085 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.354785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.378144 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:50.428532 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:50.440038 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.441269 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.441776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.449438 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.461221 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.463675 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:50.484727 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 19:58:50.526669 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:50.537328 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.538556 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.539700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.545395 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.545809 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.560998 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:50.582269 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:50.600117 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:50.624125 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:50.657040 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.658288 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.663364 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:50.664759 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:50.669714 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.675360 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.679678 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:50.712801 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:50.714136 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.717237 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.719525 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.728958 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.734195 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.747725 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:50.790822 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:50.808771 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.815229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.818341 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.818795 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:50.821086 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.825837 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:50.857165 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 19:58:50.909757 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:50.913249 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:50.914346 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.914371 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:50.927322 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:50.937556 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:50.989058 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.043968 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.048799 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.053810 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.059182 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.065055 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.065186 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.068596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.070158 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.084278 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.099282 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.102308 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.118484 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:51.183192 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:51.192006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.192315 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.195139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.196174 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.202051 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.212335 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:51.303390 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.306870 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.308373 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.312743 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:51.319782 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.319880 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.320705 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.321357 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:51.359895 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:51.397413 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.429944 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.430928 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.432375 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.436002 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.438923 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.444726 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.448410 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.458376 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.460814 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.476382 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.484380 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.490269 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:51.551724 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.576357 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.579804 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.581147 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.581756 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.601074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.603258 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:51.615741 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:51.635799 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:51.663022 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.682585 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.683277 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.683565 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.686122 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.687805 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.716540 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:51.735821 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.778757 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.781942 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.783212 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.790814 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.793834 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.794225 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:51.796240 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:51.834925 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:51.855331 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:51.856719 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:51.859858 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:51.862169 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:51.865792 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:51.870553 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:51.870895 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:52.064682 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:52.086964 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:52.092447 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.094734 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.098242 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.106410 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.109828 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:52.113794 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:52.115236 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.115717 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.191494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.192471 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.192942 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.195595 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.204074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:52.206012 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.215780 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:52.269726 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:58:52.292428 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:52.294966 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.296239 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.300069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.308331 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.313103 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.328161 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.414639 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:52.421479 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.424212 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.425511 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.432132 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.438524 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.440691 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.449301 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:52.453694 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.454697 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.464960 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.465191 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.466606 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:52.478142 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:52.483692 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:52.520069 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.564195 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.568775 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.570729 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.584532 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.585058 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.586913 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:52.626087 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.670719 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.670918 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.672147 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.672158 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.673578 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.673753 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:52.693821 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:52.695498 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:52.782927 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.783311 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.792599 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.793316 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.803925 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:52.807307 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.812722 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.813338 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.825198 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.825695 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.838632 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.841037 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:52.841164 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:52.861820 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:52.943657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:52.943856 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:52.945462 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:52.951268 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.951884 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:52.954957 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.054101 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.054741 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.060225 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.062008 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.069195 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.080530 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:53.089900 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:53.156498 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:53.165436 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.166715 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.167198 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.167346 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.168565 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:53.190353 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:53.211285 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.212792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.225123 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.225167 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.225435 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.227263 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.240056 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:53.265381 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:53.334743 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:53.354009 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.357734 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.360739 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.370905 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.374356 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.377957 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.472115 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.479804 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.484189 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.490088 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.492339 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:53.492561 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:53.513932 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:53.515818 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:53.603260 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 19:58:53.608512 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.611984 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.612967 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.618996 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.621053 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.625208 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.673835 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.674175 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.675415 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.676743 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.678510 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.697012 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:53.732415 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:53.861352 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:53.922067 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:53.922643 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:53.922899 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:53.923585 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:53.928092 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:53.935953 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:53.974041 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:53.976302 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.012698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.021893 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.022248 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.024374 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:54.025525 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.030423 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.037449 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:54.038065 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.115526 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:54.116297 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:54.117620 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.117699 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.121557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:54.124542 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.129568 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.131994 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:54.160940 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.204644 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.247781 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.256093 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.257301 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.267402 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.268215 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.268326 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.268404 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.276948 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.289093 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.293157 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.326432 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.334651 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:54.346005 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.359645 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.370100 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.413093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.415441 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.418215 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.418610 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.421379 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.426033 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:54.488382 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:54.508752 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.515569 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.517498 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.518630 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.518990 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.542157 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:54.547864 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:54.567139 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.604926 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.630290 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.633217 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.639978 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.640563 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.641287 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:54.649602 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.653836 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:54.654906 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.671262 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.678365 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.679206 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.680575 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.684034 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.698748 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.782109 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:54.784448 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.788627 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.788686 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.790506 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.805402 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:54.813444 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 19:58:54.840857 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:54.885672 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 19:58:54.886335 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:54.890246 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.893082 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:54.897239 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:54.901626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:54.915234 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:54.931838 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:54.956149 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:58:54.957580 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 19:58:55.006032 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:55.006434 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:55.007869 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:55.009754 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:55.011770 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.022372 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.026060 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.031356 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.062950 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.067239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.067620 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.078547 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.078819 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.081949 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.174522 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.175268 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.176440 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.185384 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.188944 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.196084 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:55.205322 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:55.231123 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:55.255116 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:55.271327 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.281642 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.285024 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.285884 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.291016 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.293936 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:55.322080 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:55.372704 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:55.380388 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.386408 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:55.386551 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:55.393785 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.397219 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.413370 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.431994 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.433104 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.447088 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.448489 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.455127 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.457901 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.468713 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:55.539116 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:55.561343 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:55.574798 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:58:55.578050 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 19:58:55.601074 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:55.732050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.733805 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.734607 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.734920 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.737278 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.743177 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:55.761785 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:55.766917 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:55.800762 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:55.808982 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.813163 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.816219 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.817451 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.822592 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.829800 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.907126 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:55.913190 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:55.913718 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:55.914127 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:55.914404 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:55.919220 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.026005 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.026362 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.027088 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.028383 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.038022 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.038146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:56.084192 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.098639 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.099337 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.101228 12653 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:58:56.101328 12653 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:58:56.101706 12653 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:58:56.103127 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:56.103703 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:56.103780 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
W20240501 19:58:56.104715 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 19:58:56.104851 12653 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:58:56.105947 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.111975 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.115463 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.121377 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.177573 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.177959 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.183802 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.183887 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.187320 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.188163 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.274309 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.274729 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.279357 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.283728 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:56.289942 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:56.292716 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.304852 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:56.312711 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:56.368351 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.393010 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.393709 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.395031 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.406978 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.408015 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.412253 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:56.438146 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.456621 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.461046 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.465258 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.472324 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.482199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.490203 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:56.504599 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.535938 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.536592 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.539743 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.540577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.548120 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.561751 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:56.573046 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:56.573663 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.645166 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.647786 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.653115 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.653235 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:56.655697 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.663789 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.664956 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:56.687187 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:56.753824 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:56.755213 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.758527 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.760727 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.765375 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.773159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.780342 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.826328 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.831795 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:56.835752 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.842183 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.844071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:56.856937 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:56.867187 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:56.894237 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:56.914104 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:56.916338 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:56.920888 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.921396 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:56.929291 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:56.933996 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:56.935329 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:56.958421 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.015152 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.015455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.018277 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:57.021725 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.027724 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.039232 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.039640 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.122572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.127765 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.133592 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.136090 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.136480 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.136714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.196110 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.198326 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.202364 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.209331 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.209841 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:57.211817 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.218999 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:57.279026 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:57.350286 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:57.350672 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.437583 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.438001 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.441242 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.449435 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.461905 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.461921 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:57.482448 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:58:57.509733 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.554948 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.559247 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.563118 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.563716 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.563871 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.569453 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:57.577205 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.581480 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.586011 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.593714 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.597801 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.627777 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.636739 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:57.669680 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:57.697602 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.698755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.703950 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.708819 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.709610 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.711405 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:57.734535 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:58:57.748420 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:57.749328 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.797626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.799947 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.801263 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.805336 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.809060 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.819036 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:57.821866 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:57.905676 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.909860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.913046 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.913072 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.915664 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.925429 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.971431 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:57.974658 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:57.975966 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:57.976709 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:57.978085 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:57.986199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.019119 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:58.021358 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:58:58.030529 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:58.062842 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.063279 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.064472 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.064867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.065924 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:58.089671 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:58.094710 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.141858 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:58.172820 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.178838 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.180725 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.181515 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.181922 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.183743 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.218091 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:58.292987 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.297137 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.300043 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.300460 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.315194 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.318058 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.330869 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.333416 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.343003 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.345196 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.349078 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.369498 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.443441 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.443626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.450116 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.450439 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.454136 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.467662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.546772 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.547436 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.549024 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.557264 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.559247 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.572850 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:58.575958 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.630406 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 19:58:58.669157 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.671003 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.673382 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.675843 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.678249 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:58.681258 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:58.685456 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.692121 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:58.695884 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.705469 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.705807 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:58.706494 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.707782 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:58.731858 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.732414 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:58.758370 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:58.807732 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:58.823797 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:58.824520 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.827909 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:58.828145 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:58.834988 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:58.845129 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:58:58.971870 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.004637 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.008302 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.011027 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.024612 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.027354 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.027392 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.044870 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.086266 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.088184 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.089251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.091367 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.099740 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.100121 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.113242 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.121572 13089 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:57:43.453642 13091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.453883 13091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.458604 13091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:57:44.356283 13091 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:57:44.842991 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.843185 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.843251 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.843290 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.843360 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.843396 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.843428 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.843715 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.843791 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.843843 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.843889 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.843940 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.843989 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.844033 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.844090 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:45.012038 13091 rebalancer_tool.cc:1074] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:45.077975 13091 rebalancer_tool.cc:1074] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:45.180112 13091 rebalancer_tool.cc:1074] tablet a170766f497f491ba9b45f92f2a0e751: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:45.254038 13091 rebalancer_tool.cc:1084] tablet 95b8c1a34cc94d29986289939b2159bb: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: tablet 95b8c1a34cc94d29986289939b2159bb: replica 14002a0a4c674b438c78ca5db73637cc is already present
I20240501 19:57:45.254140 13091 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:45.544260 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.544355 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544445 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.544497 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.544538 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.544593 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544644 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544710 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.544759 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.544785 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.544852 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544883 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544956 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.544997 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.545040 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.545090 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.545128 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.545153 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.545182 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.545210 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.545238 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:45.686060 13091 rebalancer_tool.cc:1074] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:45.806528 13091 rebalancer_tool.cc:1074] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:45.887768 13091 rebalancer_tool.cc:1074] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:46.059461 13091 rebalancer_tool.cc:1074] tablet 26b12620ad9a404d9ab70e8506ad1a39: '1679f84270414f67ac67c882220e1472' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:46.288522 13091 rebalancer_tool.cc:1074] tablet 6edb640b8cc04d20b0815b1beb45d9c8: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:46.424630 13091 rebalancer_tool.cc:1074] tablet 8fc8c5aab87341e3b79a8e121e6db01e: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:46.495430 13091 rebalancer_tool.cc:1074] tablet 19d8d7d255494a49920849d10b800fa7: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:46.611539 13091 rebalancer_tool.cc:1074] tablet d5faf889870246b78b82e19c041dc119: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:46.625675 13091 rebalancer_tool.cc:1084] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: tablet 2bc8fbbdc94e49d2b8b639083e083df3: replica 14002a0a4c674b438c78ca5db73637cc is already present
I20240501 19:57:46.625737 13091 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:47.001003 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.001287 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.001842 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.002202 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.002269 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.002302 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:47.002390 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.002444 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:47.002509 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:47.045264 13091 rebalancer_tool.cc:1074] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:47.057055 13091 rebalancer_tool.cc:1074] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:47.096750 13091 rebalancer_tool.cc:1074] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:47.407267 13091 rebalancer_tool.cc:1074] tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:47.415760 13091 rebalancer_tool.cc:1074] tablet 9fcd6776aebf47c99f05850de83b36e6: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:47.454274 13091 rebalancer_tool.cc:1074] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:47.481066 13091 rebalancer_tool.cc:1084] tablet 54d2db0f85ac4a90a1df779f89398463: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move ignored: Illegal state: tablet 54d2db0f85ac4a90a1df779f89398463: replica e3c6b84864cb4358a9d7e0f2a65ba83d is already present
I20240501 19:57:47.481148 13091 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:47.832376 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.832458 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:47.832499 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.832537 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.832578 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:47.832623 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.845219 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.845299 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.845373 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.845430 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:48.876021 13091 rebalancer_tool.cc:1136] tablet ee298406a4824f1aa2fa742b7dec1026: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:49.931826 13091 rebalancer_tool.cc:1136] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:51.663592 13091 rebalancer_tool.cc:1136] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:53.804268 13091 rebalancer_tool.cc:1136] tablet 9fcd6776aebf47c99f05850de83b36e6: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:54.229789 13091 rebalancer_tool.cc:1136] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.248771 13091 rebalancer_tool.cc:1136] tablet 19bfb955b9c94fdb8e0381e5b1855248: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:54.716593 13091 rebalancer_tool.cc:1074] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:54.732442 13091 rebalancer_tool.cc:1074] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:54.747366 13091 rebalancer_tool.cc:1074] tablet 6fd958b083bb430a99130a18f91acdf7: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.833585 13091 rebalancer_tool.cc:1136] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.898231 13091 rebalancer_tool.cc:1136] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:54.910290 13091 rebalancer_tool.cc:1136] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.919072 13091 rebalancer_tool.cc:1136] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:54.935956 13091 rebalancer_tool.cc:1136] tablet 26b12620ad9a404d9ab70e8506ad1a39: '1679f84270414f67ac67c882220e1472' -> '417972166f4d40ca9b4b6471a90be6af' move completed: OK
I20240501 19:57:54.956458 13091 rebalancer_tool.cc:1136] tablet 8fc8c5aab87341e3b79a8e121e6db01e: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move completed: OK
I20240501 19:57:55.078738 13091 rebalancer_tool.cc:1136] tablet a170766f497f491ba9b45f92f2a0e751: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:55.102720 13091 rebalancer_tool.cc:1136] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:55.164068 13091 rebalancer_tool.cc:1136] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move completed: OK
I20240501 19:57:55.211110 13091 rebalancer_tool.cc:1136] tablet 6edb640b8cc04d20b0815b1beb45d9c8: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:55.240355 13091 rebalancer_tool.cc:1136] tablet 19d8d7d255494a49920849d10b800fa7: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:55.251417 13091 rebalancer_tool.cc:1136] tablet d5faf889870246b78b82e19c041dc119: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:55.748528 13091 rebalancer_tool.cc:1136] tablet 5eb1f0dcd4cf49cd969eb28f1eb5d9ea: '99348cf9a56645808d00c459fb76a6f8' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:55.765035 13091 rebalancer_tool.cc:1074] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:55.779119 13091 rebalancer_tool.cc:1074] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:55.803205 13091 rebalancer_tool.cc:1074] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:55.812386 13091 rebalancer_tool.cc:1074] tablet 2d6621e1ff294dc9a474e076873cc72c: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:56.761830 13091 rebalancer_tool.cc:1136] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:56.825796 13091 rebalancer_tool.cc:1136] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:57.141567 13091 rebalancer_tool.cc:1136] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:57.179548 13091 rebalancer_tool.cc:1136] tablet 6fd958b083bb430a99130a18f91acdf7: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:57.919706 13091 rebalancer_tool.cc:1136] tablet 2d6621e1ff294dc9a474e076873cc72c: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
W20240501 19:57:58.382303 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 14002a0a4c674b438c78ca5db73637cc to server 1679f84270414f67ac67c882220e1472
W20240501 19:57:58.382398 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server ea26a64181784350bcd3540d0633b219 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:58.382468 13091 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 417972166f4d40ca9b4b6471a90be6af to server 1679f84270414f67ac67c882220e1472
I20240501 19:57:58.504098 13091 rebalancer_tool.cc:1074] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '417972166f4d40ca9b4b6471a90be6af' -> '99348cf9a56645808d00c459fb76a6f8' move scheduled
I20240501 19:57:58.542732 13091 rebalancer_tool.cc:1074] tablet 19bfb955b9c94fdb8e0381e5b1855248: 'e3c6b84864cb4358a9d7e0f2a65ba83d' -> '' move scheduled
I20240501 19:57:58.564098 13091 rebalancer_tool.cc:1074] tablet b4f0699b5bdf44499c87ff1541cd2ad3: 'ea26a64181784350bcd3540d0633b219' -> '' move scheduled
I20240501 19:57:58.881198 13091 rebalancer_tool.cc:1074] tablet 6edb640b8cc04d20b0815b1beb45d9c8: '417972166f4d40ca9b4b6471a90be6af' -> '92ecfb9f8f6745efa5aa2e168bafcc4e' move scheduled
I20240501 19:57:58.907950 13091 rebalancer_tool.cc:1074] tablet 8fc8c5aab87341e3b79a8e121e6db01e: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:58:58.928145 13091 rebalancer_tool.cc:1084] tablet d5faf889870246b78b82e19c041dc119: '14002a0a4c674b438c78ca5db73637cc' -> '417972166f4d40ca9b4b6471a90be6af' move ignored: Timed out: ChangeConfig RPC to 127.30.1.65:40027 timed out after 60.000s (SENT)
I20240501 19:58:58.928234 13091 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): unacceptable health status UNAVAILABLE
W20240501 19:58:59.131072 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.178340 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.180723 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.183266 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.183881 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.185952 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.189203 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:59.239511 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:59.259871 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:59.263306 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:59.271946 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.275269 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.282999 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.284381 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.285272 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.294209 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:59.294559 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.385222 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.386582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:59.386849 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.390645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.400498 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.401094 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.407562 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.446508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.450366 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.452819 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.453171 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.462124 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.478554 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.534597 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.534902 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.536594 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.537098 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.541230 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.550182 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.635826 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.636696 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.639124 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.641258 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.642190 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.662446 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.669003 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:59.697681 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.750718 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.753453 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.757812 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.762076 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.763679 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:58:59.773089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.783692 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:59.793979 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.798802 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.809801 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.809924 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:58:59.813551 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 19:58:59.818456 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:58:59.821781 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:58:59.837282 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:58:59.838299 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:58:59.845579 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:58:59.911593 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:58:59.912120 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:58:59.915329 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.915360 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:58:59.917734 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:58:59.925163 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:58:59.930459 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:58:59.931193 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:58:59.965008 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:59.975752 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:00.014601 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.016733 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.022590 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.025058 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.036011 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.038482 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.084374 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 19:59:00.128952 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.134454 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.138978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.143956 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.146821 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.153566 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.155553 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:00.164531 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.166230 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.171056 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.173254 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:00.193199 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:00.197989 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.198565 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.234807 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:00.274258 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:00.284968 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.285374 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.288919 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.290393 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.292382 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:00.295996 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.301023 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:00.331646 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:00.385447 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.386667 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.389780 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.391515 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.399518 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.410701 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.483916 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:00.500881 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.508283 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.508579 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.508720 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:00.509043 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.513732 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.525079 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.537189 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.538051 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:00.545570 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:00.548815 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.552747 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.555976 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.560808 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.666185 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:59:00.683504 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 19:59:00.793215 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 19:59:00.842947 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:00.844267 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:00.856622 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.862607 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.863101 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.863687 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.872886 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.882360 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.924952 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:00.927093 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:00.930080 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:00.932039 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:00.937956 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:00.938233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:00.971782 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:00.999078 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 19:59:01.028650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.029114 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.035307 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.037025 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.039070 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.044261 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.130751 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.138209 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.139544 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.144681 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.152397 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.170204 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.217094 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.226508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.232391 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.232610 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.234191 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.235541 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.236209 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:01.267549 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:01.303630 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.305203 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.314047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.316644 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.317019 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.317252 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.317854 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:01.345103 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:01.398842 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:01.402992 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.406947 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.409719 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.411206 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.413363 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.413388 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.425311 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:01.477700 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.512284 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.512728 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.513150 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.522282 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.522538 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.523794 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.530493 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.605334 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.605625 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.616660 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.618023 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.620144 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.642230 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.665238 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.667232 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.667871 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.670693 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:01.680471 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:01.681135 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.688702 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.729740 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.771922 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.776403 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.778578 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.779232 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.779982 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.787343 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.832664 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:01.882043 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.885926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.887627 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:01.893442 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.893538 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.907254 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:01.919035 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:01.936334 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:01.943396 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:01.973479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:01.974718 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:01.977170 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:01.980129 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:01.988109 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.004859 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.050288 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.052107 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.052474 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.055047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.061050 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.081070 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.113301 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:02.120730 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:02.125178 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.156476 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.158978 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.159162 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.159672 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.169181 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.180171 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.226047 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:02.257261 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.263928 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.269078 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.270080 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.279876 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.282294 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.294883 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.344231 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:02.344789 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.346657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.349004 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.355242 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.357668 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.359833 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.373438 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.473795 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:02.524284 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:02.579831 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.584494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.599300 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.601588 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.603155 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.605574 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.635118 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.689989 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.692740 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.697129 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.708163 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.712615 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.717243 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.723081 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.726029 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.728760 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:02.729804 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:02.739784 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.740898 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.749717 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:02.790074 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:02.846612 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.846882 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.848186 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.848487 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.861371 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.864347 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:02.903839 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:59:02.940773 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:02.951953 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:02.953966 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:02.954536 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:02.957244 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:02.958050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:02.959760 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.056829 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.067307 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.068367 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.084306 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.085100 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.095863 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.100129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:03.106906 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:03.107128 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.115017 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.117758 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:03.124581 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:03.126363 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.127331 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:03.138912 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:03.146253 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:03.193394 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:03.226475 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.228266 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.229874 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.231120 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.231837 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.246426 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:03.308120 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:03.349623 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.350646 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.353386 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.353796 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.361958 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.363811 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.436684 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.439854 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.442831 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.450827 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.452154 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.455454 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:03.481741 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:03.504575 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.504992 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.512240 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.513104 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.513466 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.517304 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:03.543351 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:03.606998 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.610553 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.613018 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.615763 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.621799 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.625428 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:03.642167 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:03.677529 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:03.703249 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 19:59:03.716409 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.718833 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.719739 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.722383 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.740998 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.750983 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.792218 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.793712 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.811096 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.813505 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.815390 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.817471 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:03.820354 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:03.873126 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.873773 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.875416 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.875737 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.882983 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.885643 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:03.888401 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:03.889531 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:03.962136 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:03.973670 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:03.976619 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:03.980571 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:03.996130 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:03.997076 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:03.999418 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:04.015329 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.028446 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.082727 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:04.108534 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.111512 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.113010 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.115128 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.116271 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.122781 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.290995 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.291043 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.291834 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.294550 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:04.294855 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:04.303843 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.310386 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.418809 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.423978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.425211 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.431493 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.439697 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.439733 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:04.457291 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:04.478428 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.480079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.485817 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.488000 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.491667 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.512217 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.569543 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.576161 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.579013 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.586853 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.594702 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.605074 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:04.614025 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:04.615888 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.653374 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.668887 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.672835 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:04.682021 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.686476 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.686894 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.687294 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.694118 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.694586 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:04.706015 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:04.770078 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:04.797767 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.807132 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.812255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.813575 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.819392 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.841272 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.850471 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:04.853533 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:04.853559 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:04.854532 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.873236 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.876443 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.878971 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.905661 17284 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:04.905757 17284 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:04.906106 17284 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:59:04.907442 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:04.908468 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:04.908564 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:04.909257 17284 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:59:04.945602 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 19:59:04.950891 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:04.954424 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:04.955685 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.957384 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:04.958523 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:04.962112 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:04.978243 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:05.004464 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:59:05.005131 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:05.063789 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:05.064503 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:59:05.065002 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.066565 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.073520 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.077606 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.085388 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:05.156451 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:05.178961 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.181061 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.185796 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.192111 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.194141 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.205818 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.224493 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.225055 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.228125 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.233366 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.248138 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:05.262312 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 19:59:05.263939 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:05.282035 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:59:05.288012 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:05.327986 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.331270 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.332969 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.334062 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:05.345000 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:05.345276 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:05.346561 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.349196 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.434685 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:05.435030 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:05.436136 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.438521 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.439255 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.447762 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.457983 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.532996 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:05.534503 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:05.539170 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.539608 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.541326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.555428 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:05.574501 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:05.577155 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:05.596062 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:05.599341 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.602602 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.602752 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.603997 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.607295 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.636606 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:05.689610 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:05.691210 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 19:59:05.695953 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:05.700155 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.702919 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.703491 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.704749 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.728799 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:05.767742 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:05.919183 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:05.920197 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:05.920570 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:05.921021 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:05.925788 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:05.927300 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:05.927477 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.015524 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.020660 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:06.024538 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:06.026266 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:06.031857 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:06.032342 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.032562 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.037446 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:06.072074 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:06.118620 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.122581 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.123361 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.123776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.128616 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.162257 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:06.185213 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:06.185818 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:06.237244 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.238708 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.241359 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.244699 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.265333 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.278285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.294037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.294564 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:06.300725 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:06.303961 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.305516 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.309229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.311667 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:06.314198 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:06.316326 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:06.320520 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:06.322017 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:06.374383 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:06.374969 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 19:59:06.404314 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.404347 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.405319 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.410246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.413889 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.426314 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.496461 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.500516 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.500834 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.501559 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.501987 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.517515 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.621096 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.625041 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.627245 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:06.629029 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:06.629079 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.631374 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.634069 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.638267 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.646237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.646796 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.661361 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.664997 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.692397 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.768471 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.774698 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.775178 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.775377 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.779139 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.779803 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:06.813405 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:06.879657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.880312 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:06.883883 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.891369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.891665 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.895046 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:06.984961 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:06.984989 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:06.998587 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:06.999083 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.003968 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.010345 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.037889 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.042207 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.042894 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.055487 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.059993 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.062650 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:07.079699 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.149163 14891 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:07.149291 14891 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:07.149657 14891 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:59:07.151459 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:07.153367 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:07.153431 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:07.153720 14891 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:59:07.163147 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.163317 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.166038 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.167860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.167896 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:07.170356 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.187708 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.190037 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.282930 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.285075 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:07.285207 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.286495 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:07.287784 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.288859 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.291487 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.291543 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.297385 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:07.306167 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:07.345307 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:07.365957 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:07.394177 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.415668 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.422658 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.426888 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.428187 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.435905 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.441318 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:07.451444 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:07.472563 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:07.503695 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.507400 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.513093 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.515192 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.515611 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.516659 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.745029 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.749637 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:07.751426 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:07.752705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.755246 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.765162 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.766062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.772331 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.779286 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.842957 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.845149 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.845366 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.845929 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:07.849756 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.858464 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:07.904072 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:07.935598 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:07.946672 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:07.946832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:07.950124 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.950143 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:07.952647 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:07.957147 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.058722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.065089 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.065124 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.070158 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.075233 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.075789 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:08.092525 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:08.110723 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:08.125628 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.127185 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.129155 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.131009 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.135752 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.145983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:08.167932 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 19:59:08.222344 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.222721 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.223150 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.226074 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.229480 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.234964 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:08.236196 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:08.242925 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.320843 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.323521 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.325218 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.331712 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.341984 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.342492 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:08.350286 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:08.368798 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:08.442328 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.444738 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.449757 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.454759 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.455192 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.465371 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.466296 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.468676 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.483629 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.486307 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.502282 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:08.503675 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.516475 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:08.558282 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.590492 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.596551 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.602643 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.603447 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.603878 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.620851 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:08.633275 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:08.640161 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.703809 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.704149 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.704481 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.709851 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.711328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:08.717161 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.717547 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:08.763339 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:08.817279 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:08.824211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.825127 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.826687 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.828706 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.833859 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.842664 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:08.847384 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:08.849323 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.864403 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.879607 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.888806 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.892089 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:08.914192 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:59:08.940958 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:08.967734 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.976465 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:08.976855 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:08.983664 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.985790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:08.986097 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:08.990342 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:08.991396 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.081199 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.082078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.084268 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.093151 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.093653 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.096447 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:09.161677 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:09.198887 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.201306 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.205729 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.210100 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.210233 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.210948 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.212231 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.230352 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.231832 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.243772 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.253818 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.279829 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:09.349416 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:09.552107 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:09.552860 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.553718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.560046 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.564415 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.580485 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.594588 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.620035 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.624898 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.631073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.635434 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.635964 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.636891 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:09.692700 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:09.718361 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:09.720599 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.721685 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.724622 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.731810 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.734782 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.738920 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:09.761716 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:09.814179 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:09.821332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.824785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.827775 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:09.836650 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.839072 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:09.845607 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:09.847510 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:09.879662 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:09.937152 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:09.940912 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.943728 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.944353 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:09.949393 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 19:59:09.952090 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.955304 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:09.955801 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:09.966953 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:09.977586 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.990717 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:09.991711 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:09.992481 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:09.999228 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.022806 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.031262 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:10.082880 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:59:10.098296 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:59:10.101119 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.105085 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.108207 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.115723 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.116079 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.127897 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.136881 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:10.147497 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:10.186542 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:10.206661 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.211082 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.212075 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.214246 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.222963 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.233541 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.275121 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:10.287691 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:10.313601 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.314402 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.328125 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.335003 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.344098 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.344457 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:10.347481 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:10.360729 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 19:59:10.379235 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.383244 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.384325 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.384896 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.385372 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.385856 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.477810 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.478992 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.482595 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:10.484802 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:10.489187 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.491937 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.492729 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.578882 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:10.589110 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.590499 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.594861 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.595752 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:10.596518 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:10.613867 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.622956 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.633867 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:10.670142 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.671310 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.675359 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.683776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:10.684203 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:10.686710 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.711802 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.718016 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 19:59:10.743325 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.748262 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.751991 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.752445 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.765573 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.767244 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:10.822264 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:59:10.827267 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:10.849752 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.851069 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.853579 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.854142 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.854910 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.881172 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:10.908895 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:10.955045 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:10.961063 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:10.963255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:10.965438 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:10.972244 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:10.978829 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.036414 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.041376 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.043040 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.043829 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.050105 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:11.057448 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:11.065936 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:11.075332 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:11.098201 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:11.230507 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:11.279454 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:11.282241 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:11.283917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.285642 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.292405 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.295074 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.304508 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.308246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:11.320188 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 19:59:11.396450 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.396616 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.397626 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.399554 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.410606 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.418556 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.421756 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.431699 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.431787 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.438891 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:11.450718 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:11.453838 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:11.459745 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:11.477684 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:11.499516 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:11.517448 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 19:59:11.555122 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.556241 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.556767 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.559512 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.560704 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.569175 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.665493 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.665565 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.668627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.676528 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.685658 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.693171 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:11.757010 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:11.759358 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.761196 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.765672 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.766000 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.767835 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.767840 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.826432 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.827283 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.830338 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.833043 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.833398 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.848089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:11.884826 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:11.926071 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:11.927474 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:11.929546 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:11.933398 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:11.941890 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:11.958076 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.054855 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.060516 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.072317 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.074456 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.078460 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.087824 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.111832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.118968 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.119866 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.129089 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:12.136947 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:12.139487 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.141093 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:12.161003 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.167346 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:12.219990 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.222010 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.227438 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.228509 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.229907 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:12.239420 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:12.249965 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.327124 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.329615 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.333997 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.337400 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.338833 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.349383 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:12.355365 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.369325 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.380508 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.394178 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:12.433621 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.433641 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.440316 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.450256 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.474207 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.480206 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.489267 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.489835 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.494118 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.501055 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.503526 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.507778 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:12.523451 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.561159 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:12.599843 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.609208 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.609437 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.610100 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.610358 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.610464 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:12.668192 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:12.701666 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:12.704327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.706357 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.717626 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.717803 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.719178 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:12.735258 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.747676 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:12.798986 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:12.799242 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:12.801071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:12.802317 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:12.804289 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:12.817130 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:12.821419 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:12.962484 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:12.970364 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:13.003064 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.007794 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.009910 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.013193 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:13.013451 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:13.027398 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.032181 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:13.037969 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:13.076658 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 19:59:13.126503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.130836 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.131511 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.136375 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.141470 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.147989 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.151019 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.152765 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.153286 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.157061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:13.161824 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:13.167161 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:13.181186 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.201681 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.270920 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.272082 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.277465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.282919 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.288895 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.293188 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.378418 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.382535 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.382575 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.385104 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.385314 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.388626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:13.406720 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:13.432855 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:13.495923 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.497316 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.498836 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.503603 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.506826 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.508775 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.521731 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.522310 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.523979 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.534905 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.539372 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.553601 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:13.561715 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:13.564890 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:13.578615 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:13.628098 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:13.640623 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.641363 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.643872 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.646234 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.654246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.662247 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:13.662423 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:59:13.708314 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:13.723752 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:13.750656 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.754848 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.755255 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.755932 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.757664 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.758289 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:13.812402 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:13.873778 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.873847 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.878701 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.880327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.899677 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.901103 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:13.902619 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:13.905522 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.918185 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:13.918390 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:13.919374 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:13.932950 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:14.001338 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:14.007161 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:14.025023 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.028373 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.034425 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.037081 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.037731 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.040953 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:14.064087 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:14.088330 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:14.140468 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.143522 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.144465 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.145656 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.146862 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.147034 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.250133 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.259433 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.259501 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.260726 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.261056 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.270318 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.298384 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.298709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.299778 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.304867 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.318043 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.325007 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.409813 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.411878 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.416458 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:14.418319 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:14.425278 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.434861 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.446187 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:14.498525 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:14.565127 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.570770 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.571303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.573551 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.577047 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.577601 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:14.621100 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:14.671061 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:14.671700 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:14.697791 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:14.729987 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.734722 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.738749 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.741061 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.741948 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.749641 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:14.803344 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:14.830960 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.831559 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.832625 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.833622 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.845325 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.864655 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.928714 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.933063 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:14.934995 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.936861 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.937793 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.945436 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.990262 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:14.993184 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:14.997922 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:14.998210 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:14.999583 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.001549 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:15.018170 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 19:59:15.040632 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:15.043903 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:15.046137 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:15.063462 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:15.108527 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.109835 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.110226 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.111495 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.117550 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.118121 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.154069 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:15.211745 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 19:59:15.213881 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:15.219480 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.219735 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:15.221697 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:15.225139 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.225622 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.226233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.233505 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.237282 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:15.249035 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:15.291899 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.293493 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.294507 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.295254 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.298954 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.309803 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.322031 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:15.379882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:15.382354 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 19:59:15.384274 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.384805 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:15.387526 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:15.388186 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.391741 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.393186 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.407886 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.483618 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:15.483871 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:59:15.486635 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:15.488709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.488718 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.489917 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.491292 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:15.494640 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:15.509225 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.523126 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 19:59:15.605278 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.607074 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.610121 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.624492 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.625878 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.628809 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.635408 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:15.645159 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:15.645947 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.648417 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.659724 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.661087 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.681963 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.746985 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 19:59:15.751509 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.751711 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.756999 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.757270 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.768568 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.768898 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.856850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.856933 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.858690 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.860116 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.870626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.881361 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:15.958153 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:15.963737 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:15.966920 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:15.969205 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:15.981993 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:15.982559 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:15.984012 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:15.990281 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:16.007036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:16.013414 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:16.017077 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.017887 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.018615 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:16.029821 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:16.029946 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:16.036087 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.036091 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.129755 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.132326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.136584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.145104 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:16.148344 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:16.153232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.169339 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:16.224696 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:16.243783 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:16.345306 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.345997 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:16.348889 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 19:59:16.351351 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.356328 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.356995 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.364857 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.448433 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.448575 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.449774 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.450146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.451992 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.457588 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:16.496989 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:16.527868 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:16.535691 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:59:16.543896 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.547528 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.548398 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.553617 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.553876 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.556546 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:16.583758 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:16.628578 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:16.634918 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:16.665195 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.672570 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.680732 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.682400 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.683395 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.703662 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.704144 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.704627 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.731146 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.732924 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.738643 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.740074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.822474 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.828903 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.829258 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.830770 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.833752 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.849584 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:16.923734 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:16.924849 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:16.925951 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:16.926434 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:16.930408 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:16.938143 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:16.957010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:16.974151 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:17.036551 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.042186 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.045054 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.056581 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:17.058681 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.081282 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.087940 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.088758 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.095304 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.100181 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.114802 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:17.129309 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.129616 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:17.146003 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:17.152848 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:17.205789 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.213829 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.214417 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.215039 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.218254 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.218915 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:17.227015 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.316792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.318316 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.324414 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.326131 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.327693 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.337714 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:17.346809 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:17.410061 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.422776 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.425105 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.427664 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.434185 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.448585 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:17.456434 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:17.458698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.464232 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.465658 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.467303 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.467371 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:17.472369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.485338 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:17.486178 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.574288 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.577859 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:17.578755 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:17.582255 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.582348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.590008 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.605491 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.605585 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.679232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.679522 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.681087 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.686056 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:17.693521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:17.701719 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:17.705961 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:17.714429 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:17.742554 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:17.766197 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.782645 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.785213 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.786244 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.789078 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.797621 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:17.799264 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:17.813439 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:17.816231 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.839569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:17.840164 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:17.858501 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:17.859449 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:17.866891 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:17.868059 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:17.884136 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:17.956625 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:18.094389 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:18.116577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.120663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.123479 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.126235 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.127434 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.145242 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.196708 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.198206 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.199510 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.202006 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.202556 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.211437 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:18.224996 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:18.291458 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.295497 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.295722 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.296140 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.296315 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.305704 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:18.333425 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:18.349529 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:18.398273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.400418 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.415377 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.416322 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.421046 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.433933 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.468714 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.472465 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.474915 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.476090 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.478941 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.488265 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.555886 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.556560 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.563545 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.565315 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.566215 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.589443 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:18.596341 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:18.665454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.665807 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.666180 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.667559 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.668331 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.670030 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:18.683151 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:18.700945 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:18.752861 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 19:59:18.786094 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.790787 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.797106 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.798981 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.802333 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.807312 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.808749 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.823954 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.829977 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.833467 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.833797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:18.838172 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:18.887210 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:18.887279 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:18.934803 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:18.935075 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:18.937101 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:18.939620 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:18.940830 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:18.948741 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:18.950505 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:19.012009 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:59:19.019146 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:19.025230 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:19.046286 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.046694 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.057722 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.058346 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.059729 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.076767 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.150920 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:19.155437 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:19.158843 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.164924 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.165053 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.181546 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.187494 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.197345 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.202790 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.216367 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.217017 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.226712 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.237506 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:19.261512 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:19.314087 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.317194 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.318830 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.321746 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.326000 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.341192 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:19.364774 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:19.416254 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.418267 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.421350 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.422747 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.425158 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.430081 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.517323 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.520233 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.520287 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.528355 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.529234 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.539515 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:19.553912 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:19.565392 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:19.569793 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.570068 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.585996 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.602085 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:19.610680 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:19.612548 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:19.692811 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:19.693431 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:19.854483 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:19.888653 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.891140 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.896785 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.905555 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:19.906641 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:19.912518 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.922572 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:19.936414 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:19.939725 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:19.941952 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.958294 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:19.958717 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:19.970655 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:19.971565 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:19.971975 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:20.006815 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:20.032267 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:20.049304 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.051008 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.053848 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.055531 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.056092 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:20.065609 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 19:59:20.077623 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:20.120623 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:20.149446 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:20.151938 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:20.159687 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:20.162952 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:20.163991 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.165982 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.172072 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.177641 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:20.224176 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:20.260231 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.261049 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.264334 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.266513 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.266559 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:20.267331 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 19:59:20.287165 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:20.304364 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.311968 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.312031 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.317080 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.318830 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.329316 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.333175 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:20.355113 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:20.357276 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:20.415459 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.415724 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.416704 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.420104 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.424018 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.434796 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:20.442042 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:20.448556 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 19:59:20.473912 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:20.513142 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.513867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.514577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.515734 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.518888 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.539503 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:20.592864 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 19:59:20.637529 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.638451 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.640729 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.648594 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:20.649413 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:20.651651 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.653648 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.656818 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.661944 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.671236 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.683369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.687350 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.705315 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:20.744010 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:20.778152 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.781790 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.783905 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.789392 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.797075 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.801662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:20.845640 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240501 19:59:20.881193 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:20.881671 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:20.893349 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:20.894632 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.900813 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:20.914476 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:20.916383 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:20.953538 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:20.991128 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:20.994779 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.000521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.004664 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.012796 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:21.024394 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:21.024919 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.060072 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.061990 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.062418 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.065073 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.066818 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.066866 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.157480 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.158443 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.158634 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.165369 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.173624 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.181192 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.260226 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.260643 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.262645 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.263799 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.268939 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.282474 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:21.304240 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:21.309028 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:21.313094 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:21.339337 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 19:59:21.360307 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.362192 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.366559 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.367830 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.372753 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.373600 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:21.492065 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:21.535163 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:21.562893 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.564041 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.564590 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.564774 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.572912 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.579497 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:21.618485 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:59:21.661629 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:21.676685 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:21.682071 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:59:21.683874 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.685716 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.685933 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.688094 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.694232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.696218 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.707304 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.728579 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.734908 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.739816 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.744643 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.745199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:21.785708 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:21.828732 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.832551 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.833257 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.836328 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.837194 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.838560 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.842914 20753 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:21.843014 20753 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:21.843384 20753 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:59:21.848594 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:21.848701 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:21.848766 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:21.848937 20753 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 19:59:21.892283 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:21.912719 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:21.937819 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:21.944562 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:21.944604 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:21.944697 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:21.947424 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:21.961198 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:21.977988 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.055903 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.062386 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.063799 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.069190 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.072264 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.086571 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.087322 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.093940 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.101527 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.104364 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.110898 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.128541 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:22.157999 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.212044 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.217072 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.217082 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.224185 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.226150 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.228783 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:22.235765 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:22.317649 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.321815 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.325007 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.325078 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.329667 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.330961 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:22.338081 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:22.345247 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:22.362432 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:22.397300 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:22.417184 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.430982 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.433192 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.435268 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.446190 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.454594 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.459477 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.464494 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.466187 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.470115 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.478826 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.480454 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.514637 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:22.550617 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.589735 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.591153 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.603655 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:22.606060 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.607991 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.612109 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.613044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:22.628044 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:22.655012 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.694278 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.695583 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.703503 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.703670 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.707592 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.733044 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:22.746138 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.790931 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.791373 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.792737 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.801066 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.804307 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:22.806444 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:22.806772 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.858865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:22.860133 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.863035 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.866294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.868212 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.871192 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:22.877470 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.881243 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:22.888170 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:59:22.891578 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:22.963609 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:22.965237 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:22.967083 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:22.967449 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:22.977782 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:22.984514 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:23.103873 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:23.106181 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.108357 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.113374 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.115706 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.123486 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.128099 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:23.139966 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:23.249526 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.267119 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.267421 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.269167 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.277824 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.287819 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:23.299124 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.299224 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:23.330507 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:23.381058 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.382347 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.383327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.386370 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:23.389140 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.395828 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.396174 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.445232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.447798 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.459460 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.463086 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.463788 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.472385 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:23.489722 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:23.542027 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.544788 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.547405 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.547941 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.547999 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.557850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:23.628607 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:23.650676 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.655907 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.658006 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.658303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.669416 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.685243 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:23.757737 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.769166 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.773837 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.775465 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.780418 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.784931 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.789783 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.789952 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:23.793594 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.797080 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.803495 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.825731 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.835172 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:23.845011 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:23.897542 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:23.918655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:23.919785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:23.920965 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:23.932533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:23.933218 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:23.934012 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:23.942131 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.023166 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.026542 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.028009 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.029287 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.046854 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.059790 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.140712 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.144455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.145912 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.146013 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.146389 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.153354 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:24.162236 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:24.174535 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 19:59:24.195569 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.200639 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.204947 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.209496 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.209964 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.211377 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.311213 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.313323 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.316277 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:24.318315 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:24.326403 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.329025 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.333055 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:24.336828 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:24.359405 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:24.427938 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.432478 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.433141 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.440112 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.449231 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.461504 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.505903 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.507771 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.517159 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.520038 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:24.531908 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:24.533295 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.547717 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:24.550161 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:24.593580 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:24.604609 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.606221 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.607055 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.613083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.626842 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.635495 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:24.637579 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:24.709417 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.710132 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.713117 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.715945 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.724081 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.724309 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:24.789532 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:24.831790 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:24.832352 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:24.833351 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:24.834746 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:24.834759 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:24.853739 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:24.891713 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:24.950049 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:24.962651 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:24.997699 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:25.011147 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.018831 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.018945 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.019486 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:25.022126 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.026167 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.034116 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:25.034216 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:25.091711 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 19:59:25.092402 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.119598 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.124096 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.129222 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.129465 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.129796 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.131085 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:25.153966 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:25.190834 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.191720 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.195394 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.198742 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.199079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.202159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.265911 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.266925 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.270596 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.270989 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.273658 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.273921 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:25.286754 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:25.301569 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 19:59:25.376951 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.377045 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.379031 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.388604 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.390434 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.409255 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:25.453704 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:25.498828 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:25.500101 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.502153 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:25.508499 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.511173 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.511904 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.512226 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.516691 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.530874 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:25.532119 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:25.532773 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.554900 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.556926 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.557081 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.557336 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:25.581471 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:59:25.581553 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:25.593025 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.640174 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.643970 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.644482 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.646591 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.654294 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.656329 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:25.668507 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 19:59:25.673314 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:25.744987 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.745905 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.746796 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.747058 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.747759 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.747923 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:25.755910 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:25.777210 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.847924 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.849617 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.859242 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:25.864343 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.870899 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:25.871064 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:25.880431 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:25.923557 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 19:59:25.925690 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:25.928858 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:25.929309 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:25.931496 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:25.936504 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.938336 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:25.967926 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.000859 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.001631 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.006721 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.007449 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.013376 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.015157 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:26.030509 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:26.126143 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.127007 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.127468 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.128383 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.140633 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.145184 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.232656 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.236894 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.237793 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.240613 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.246778 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.247906 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:26.279840 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:26.310940 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.311506 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.320081 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.328764 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.333986 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.344127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.393285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.395797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.396832 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.397398 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.401063 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:26.405894 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:26.409844 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:26.410816 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:26.502348 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:26.558370 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:26.627167 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.627552 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.635278 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.636654 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.636678 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.637269 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:26.686354 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:26.703850 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:26.717460 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.719568 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.719689 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.721544 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.722872 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.737856 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:26.767001 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:26.806128 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:26.813365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.815960 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:26.817095 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 19:59:26.822713 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.829434 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.829591 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.838622 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:26.844800 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:26.909443 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:59:26.925794 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:26.930032 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:26.935092 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.945128 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:26.946341 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:26.948421 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:26.995501 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.003152 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.003329 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.006111 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.021471 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.021647 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:27.107357 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.151789 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.153971 25604 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:27.154068 25604 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:27.154456 25604 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 19:59:27.154767 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.156330 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:27.156841 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:27.156934 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:27.159652 25604 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:59:27.159775 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.161396 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.165302 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.181454 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.186236 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:27.238422 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:27.254873 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:27.273769 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:27.285238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.285661 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.286522 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.287461 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.297197 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:27.299079 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:27.299157 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.309461 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.372836 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.375682 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.378999 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.380136 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.385712 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:27.399139 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:27.401815 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.441462 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.442404 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.447175 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.449734 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.453272 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.463209 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:27.517428 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.541323 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.545235 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.547509 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.552264 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.552697 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.557135 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.597678 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.655906 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.656809 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.657510 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.669513 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.669956 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.678673 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.711856 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:27.730883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.732333 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.739769 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.744498 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:27.746836 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.747566 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.765254 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.773269 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:27.812752 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.812752 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.813675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.814702 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.814889 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.817508 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:27.854620 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:27.908304 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:27.916546 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:27.917764 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:27.921759 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:27.922935 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:27.922973 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:27.933727 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:27.972523 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:27.973976 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:28.010082 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.012977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.017102 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.028264 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.035638 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.049549 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:28.073256 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:28.095651 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.097273 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.097455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:28.105959 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:28.106730 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.107213 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.111560 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:28.159121 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:59:28.163532 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:28.177386 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:28.184245 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.186324 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.186343 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.186599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.189249 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.197196 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:28.303557 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:28.397612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.398648 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.401288 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.401846 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.408855 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.431555 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:28.463204 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:28.486340 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.487226 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.489723 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.493971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.501366 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.507611 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:28.572048 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:28.587167 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.588038 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.596589 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.597894 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.598143 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.611793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:28.634263 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:28.697417 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.701020 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.708084 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.711336 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.712785 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.738981 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:28.750000 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:28.762853 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.763484 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.764608 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.774703 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.777073 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.781844 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.853760 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.859061 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.863620 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.865525 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.871623 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.873776 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:28.926275 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:28.961747 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:28.962265 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:28.963670 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:28.970256 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:28.977226 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:28.983423 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.027990 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:29.057511 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:29.058017 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.076503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.077531 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.079756 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.086268 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.087741 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.087751 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.106443 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.141160 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.141813 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.141932 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.145663 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.151295 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.171097 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.226764 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.230383 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.231431 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.232141 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.233194 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.249157 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:29.251521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.331833 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 19:59:29.334796 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.335348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.340808 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.346211 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.349897 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.356034 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:29.407893 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:29.452612 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.458020 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.468884 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.469514 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.472988 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.480623 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:29.482203 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.497833 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.500195 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.501150 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.503192 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.531435 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.532346 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:29.549055 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.600662 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.605091 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.607007 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.607095 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.618080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.622169 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.647590 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:29.696261 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.698017 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.702032 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.706538 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.713171 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.714352 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.733186 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:29.802568 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.803933 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.812017 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:29.815006 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.828940 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.833395 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.835533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:29.853946 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:29.874671 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.875442 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:29.877246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:29.882237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:29.884135 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:29.886106 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:29.918637 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:29.940438 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:30.027328 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:30.055647 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:30.124114 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.124369 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.128845 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.130016 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.133535 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.141356 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:30.201962 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 19:59:30.207082 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.210129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.213884 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.216182 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.216215 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:30.242230 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:30.244410 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.285550 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.286352 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.288971 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.294728 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.301007 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.312999 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:30.323285 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:30.355679 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:30.361855 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:30.366614 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:30.392678 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 19:59:30.396867 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:30.400282 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.400679 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.401517 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.401605 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.404966 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.417832 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.499624 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.500586 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:30.502311 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:30.504794 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.511222 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.516572 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.527992 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:30.540580 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:30.542349 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:30.571543 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:30.574882 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:30.576155 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.582628 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:30.595417 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:30.606592 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.606801 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.619724 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:30.649238 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:30.655006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.657029 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.657693 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.658519 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.668918 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.673913 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:30.676964 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 19:59:30.715693 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:30.755453 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.756035 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.756290 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.759263 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.759737 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.761942 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:30.806103 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:59:30.851498 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:59:30.855129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.857405 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.857692 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.863703 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.863749 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.864477 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.943223 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:30.947587 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:30.949905 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:30.974753 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:30.975327 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:30.977067 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.003268 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.017163 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.019994 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.025067 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.038583 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:31.039904 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 19:59:31.049175 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:31.080793 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:31.090773 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:31.113749 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:31.130776 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.131480 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.134513 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.137089 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.137327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.141939 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:31.166684 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:31.236922 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.244169 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.246650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.251241 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.257164 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.261061 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:31.313253 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:31.329360 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.333439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.334026 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.336143 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.346796 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.347049 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.397397 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.399530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.403020 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.403895 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.405270 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.410929 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.495072 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.497256 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.499090 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.500794 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.506875 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.510159 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:31.510181 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:31.572607 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:31.609573 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:31.627521 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:31.641958 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.646556 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.648272 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.650326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.650468 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.658241 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:31.689339 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:31.799152 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.800668 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.801117 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:31.801119 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.801349 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.816439 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:31.822161 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 19:59:31.894233 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:31.897797 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:31.899382 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.902575 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:31.909055 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:31.920040 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:31.980829 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 19:59:32.001415 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.007109 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.008405 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.035028 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:32.035408 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:32.039770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.043983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.048547 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:32.049635 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:32.055210 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.056413 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.058248 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.070103 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.080647 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:32.118579 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:32.175696 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:32.178246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.178481 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.180233 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.182412 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.183977 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.184543 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:32.218327 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:32.233983 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:32.273514 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.277849 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.280027 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.281356 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:32.284327 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:32.289180 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.297197 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.393316 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.394356 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.406116 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.411535 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.413319 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.413789 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.414094 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.421041 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.422020 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.426946 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.443770 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.455794 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:32.496594 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:32.533656 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:32.535832 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.536389 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.540853 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.541064 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.560791 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.565008 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.650709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.652359 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.656775 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.659080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.664163 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.665038 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.747452 10830 maintenance_manager.cc:419] P 06cf8280f20843729e99a32633f62d20: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033387
I20240501 19:59:32.761029 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.761682 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.771901 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.774053 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.786043 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.789266 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.794514 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.799036 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.800817 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:32.801046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:32.802656 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:32.803586 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:32.807973 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:32.830135 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.833756 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:32.841090 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:32.844249 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:32.855316 10829 maintenance_manager.cc:643] P 06cf8280f20843729e99a32633f62d20: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.017s sys 0.004s Metrics: {"bytes_written":20359,"cfile_init":1,"dirs.queue_time_us":11239,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":10146,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":117,"threads_started":2,"wal-append.queue_time_us":24222}
I20240501 19:59:32.856284 10830 maintenance_manager.cc:419] P 06cf8280f20843729e99a32633f62d20: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7603 bytes on disk
I20240501 19:59:32.856865 10829 maintenance_manager.cc:643] P 06cf8280f20843729e99a32633f62d20: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 19:59:32.924862 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.925277 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:32.927129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:32.933282 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:32.941282 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:32.941656 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:32.958993 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:32.962023 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:32.962778 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:33.001371 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:33.026341 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:33.027729 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:33.032125 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.032924 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.037492 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.048873 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.050920 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:33.062281 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:33.104544 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:59:33.111006 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:33.135161 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.135550 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.136766 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.142206 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.153071 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.154114 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.207510 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.208103 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:33.211091 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:33.215390 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.216579 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.219415 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.223031 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:33.245991 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:33.265220 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:33.459717 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.463882 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.465250 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.473253 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.477404 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.490473 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.543586 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.544409 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.550370 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.550541 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.554595 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.571579 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.649935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.657716 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.658493 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.658560 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.659516 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.665400 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:33.672952 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:33.678073 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:33.778470 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.780467 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.781266 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.782685 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.799010 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:33.799317 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:33.801519 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.805069 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.806068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:33.827849 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:33.828882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.833734 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:33.836222 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:33.837751 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.844820 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.930142 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:33.932058 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:33.934082 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:33.935536 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:33.939705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:33.953209 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.038848 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.039769 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:34.043968 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.045208 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.049816 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.052546 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.054909 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.147524 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.152724 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.154217 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:34.162603 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.165428 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.183647 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.190589 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.193634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:34.200544 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 19:59:34.210875 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:34.211902 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.213188 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.219290 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.221297 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.232334 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:34.253360 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:34.308418 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.311439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.313642 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.317706 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.325660 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.326078 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:34.334726 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:34.400882 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:34.414508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.418076 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.419063 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.428225 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.429306 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.438172 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:34.477443 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.520159 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.523618 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.527323 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.527433 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.534832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.536396 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.591059 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.603245 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.607554 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.607935 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.614648 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.625361 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.702572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.703622 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.703747 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.718040 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.718267 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.718310 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:34.757664 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:34.782130 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.801311 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.803552 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.805878 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.814457 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.820688 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.836402 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:34.862318 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:34.900384 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.907778 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.908128 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.919253 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:34.922188 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.935420 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:34.943845 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:34.962282 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.966604 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:34.975855 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:34.977303 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:34.977488 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:34.981943 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:34.987943 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:34.994328 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.009850 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:35.033514 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.206068 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 19:59:35.228863 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.232333 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.234094 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.242465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.245273 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:35.247421 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.263715 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:35.292776 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:35.307687 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.308367 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.314646 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:35.331355 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.334729 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.344965 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.345047 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.346618 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.347332 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:35.433437 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.433614 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.433898 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:35.435451 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:35.436604 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.438064 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.439085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.447741 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.461839 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.465665 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 19:59:35.468250 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.520677 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:35.544229 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.545105 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:35.546913 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.551393 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.551834 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.558585 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.593726 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:35.620080 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.622653 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.625126 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:35.641858 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.641979 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.650998 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.680655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.694291 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:35.695286 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.695338 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.699679 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.717370 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.787897 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.790393 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.790985 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.794099 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.794656 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.803376 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 19:59:35.805055 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:35.823400 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:59:35.847429 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:35.864132 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:35.875404 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:59:35.892776 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.895493 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:35.895871 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:35.896963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:35.900271 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:35.917475 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:35.921684 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:35.993304 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:35.998296 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.002384 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.005578 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.018914 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.036208 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.073693 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.077275 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.078342 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.079046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.080168 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.089766 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.180649 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.184042 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:36.184856 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:36.186028 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.188038 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.194757 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.205425 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:36.205534 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 19:59:36.269964 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:36.289506 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.290556 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.293396 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.299376 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.306511 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.307179 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:36.324064 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:36.366472 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.367034 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.369665 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.372790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.373487 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.406895 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:36.437340 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:36.441222 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.441452 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.447326 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.456315 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.456364 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.460400 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:36.462636 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:36.496857 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:36.517390 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:36.537245 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.537940 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.542265 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.550451 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.552039 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.570886 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:36.594401 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:36.608697 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 19:59:36.653685 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.656417 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.661087 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.663069 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.663297 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.669991 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:36.722663 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:36.728819 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.729956 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.730530 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.734755 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.734815 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.745095 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:36.926404 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:36.955521 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:36.966161 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:36.971438 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:36.974167 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:36.974524 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:36.983327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:36.986778 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.075714 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.079757 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:37.086477 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:37.088311 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.106629 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.114360 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.118127 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.127315 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.128935 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.132562 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.141798 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.144515 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.167614 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.244866 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.246577 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.249539 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.254676 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.254680 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.278414 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:37.280799 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:37.293042 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:37.349402 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:37.359508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.364362 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.364389 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.369933 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.373241 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.374593 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.470803 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.473255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.477905 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.479988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.481289 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.484153 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.486546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.490599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.509922 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.510179 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.515386 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.535250 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:37.540630 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:37.563089 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:37.571928 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:37.584744 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:37.626161 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.627941 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.628165 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.631115 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.632388 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.645965 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:37.660549 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:37.667059 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:37.667062 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:37.727361 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:37.728207 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:37.728317 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.733312 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.737701 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.740628 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.752658 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:37.764521 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:37.781786 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:37.831135 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.831324 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.832525 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.836700 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.841516 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.855360 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.882520 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:37.886431 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:37.897521 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:37.898437 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:37.898785 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.900033 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:37.924216 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(8fc8c5aab87341e3b79a8e121e6db01e): perf score=0.033376
I20240501 19:59:37.957640 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(8fc8c5aab87341e3b79a8e121e6db01e) complete. Timing: real 0.033s user 0.030s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":146,"dirs.queue_time_us":4274,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":863,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":186,"threads_started":3,"wal-append.queue_time_us":161}
I20240501 19:59:37.958915 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8): perf score=0.033384
I20240501 19:59:37.975111 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":110}
I20240501 19:59:37.975889 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(8fc8c5aab87341e3b79a8e121e6db01e): 4834 bytes on disk
I20240501 19:59:37.976680 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(8fc8c5aab87341e3b79a8e121e6db01e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
I20240501 19:59:37.977421 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8): 4581 bytes on disk
I20240501 19:59:37.977699 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.007527 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.013517 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(8fc8c5aab87341e3b79a8e121e6db01e): perf score=0.033402
W20240501 19:59:38.014979 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:38.016211 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.018303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.033258 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.040954 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.058892 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(8fc8c5aab87341e3b79a8e121e6db01e) complete. Timing: real 0.045s user 0.022s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":2992,"dirs.queue_time_us":10198,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":208,"threads_started":3,"wal-append.queue_time_us":2728}
I20240501 19:59:38.060024 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(8fc8c5aab87341e3b79a8e121e6db01e): 4834 bytes on disk
I20240501 19:59:38.060678 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(8fc8c5aab87341e3b79a8e121e6db01e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.061065 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.061409 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(19f79cd7d4ec4b4697a2637e6d14a892): perf score=0.033414
W20240501 19:59:38.069020 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:38.073331 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:38.073905 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:38.086861 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(19f79cd7d4ec4b4697a2637e6d14a892) complete. Timing: real 0.025s user 0.019s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":4617}
I20240501 19:59:38.087677 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892): 4921 bytes on disk
I20240501 19:59:38.088027 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 19:59:38.100942 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:38.144107 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:38.146718 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:38.250803 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.251475 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.256892 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.267824 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.269181 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.270445 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:38.277369 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:38.302795 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.303454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.305073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.305539 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.315277 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.323854 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:38.341578 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:38.367079 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 19:59:38.406327 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.406702 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.410499 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6): perf score=0.033347
I20240501 19:59:38.413903 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.418102 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.419462 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.444993 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.460649 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.050s user 0.024s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":171,"dirs.queue_time_us":6376,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":813,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":191,"threads_started":3,"wal-append.queue_time_us":260}
I20240501 19:59:38.461742 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6): 4834 bytes on disk
I20240501 19:59:38.462249 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.479642 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(95b8c1a34cc94d29986289939b2159bb): perf score=0.033368
I20240501 19:59:38.495764 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(95b8c1a34cc94d29986289939b2159bb) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1196,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":140}
I20240501 19:59:38.496546 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(19d8d7d255494a49920849d10b800fa7): perf score=0.033372
I20240501 19:59:38.515893 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.019s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1116,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":273}
I20240501 19:59:38.516541 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7): 4921 bytes on disk
I20240501 19:59:38.518574 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.519254 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb): 4581 bytes on disk
I20240501 19:59:38.519970 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240501 19:59:38.534324 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:38.535267 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.540380 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.541112 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.541438 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.558015 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:38.558151 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:38.569662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:38.623543 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:38.660006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.662204 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.667488 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.668084 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.691311 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.693238 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.839309 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248): perf score=0.033336
I20240501 19:59:38.840473 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(8e48d711c5714ee58ebbcbd094ebf3e7): perf score=0.033336
I20240501 19:59:38.858528 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(8e48d711c5714ee58ebbcbd094ebf3e7) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118,"threads_started":2,"wal-append.queue_time_us":444}
I20240501 19:59:38.859284 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(8e48d711c5714ee58ebbcbd094ebf3e7): 4581 bytes on disk
I20240501 19:59:38.859611 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(8e48d711c5714ee58ebbcbd094ebf3e7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.870628 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.031s user 0.020s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":158,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1084,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":206,"threads_started":3,"wal-append.queue_time_us":113}
I20240501 19:59:38.871619 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248): 4834 bytes on disk
I20240501 19:59:38.872081 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 19:59:38.921347 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:38.927929 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:38.928017 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.930786 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:38.938284 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:38.951995 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:38.952744 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:38.963985 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3): perf score=0.033372
W20240501 19:59:38.969861 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:38.993084 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.029s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1387,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2208,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":117,"threads_started":2,"wal-append.queue_time_us":244}
I20240501 19:59:38.993862 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3): 4921 bytes on disk
I20240501 19:59:38.994236 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 19:59:38.994943 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(8e48d711c5714ee58ebbcbd094ebf3e7): perf score=0.033377
I20240501 19:59:39.014580 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(8e48d711c5714ee58ebbcbd094ebf3e7) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":1172,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":3213}
I20240501 19:59:39.015487 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(8e48d711c5714ee58ebbcbd094ebf3e7): 4581 bytes on disk
I20240501 19:59:39.016062 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(8e48d711c5714ee58ebbcbd094ebf3e7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240501 19:59:39.023535 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:39.039678 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:59:39.051275 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.054881 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:39.056834 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.071182 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.075292 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:39.078910 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.080950 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.096319 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:39.100210 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.120787 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.133702 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.140250 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.144039 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.151674 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.154189 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.161832 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:39.236310 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.246758 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.249250 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.251266 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.252377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.254379 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.254715 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.351992 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.352813 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.361326 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.361655 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.366796 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.371148 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.373884 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(c49b9c5ebaab42e18fe7bbe8127d569e): perf score=0.033335
I20240501 19:59:39.394228 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(c49b9c5ebaab42e18fe7bbe8127d569e) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1586,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":126}
I20240501 19:59:39.395193 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3): perf score=0.033340
I20240501 19:59:39.410365 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":352}
I20240501 19:59:39.410954 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(c49b9c5ebaab42e18fe7bbe8127d569e): 4921 bytes on disk
I20240501 19:59:39.411259 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(c49b9c5ebaab42e18fe7bbe8127d569e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 19:59:39.411723 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3): 4834 bytes on disk
I20240501 19:59:39.412055 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 19:59:39.426036 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:39.473574 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.474831 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.475613 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.483757 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.486052 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.486948 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.489596 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.490206 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.495169 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.504010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.518067 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(a170766f497f491ba9b45f92f2a0e751): perf score=0.033375
I20240501 19:59:39.522902 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3): perf score=0.033377
I20240501 19:59:39.527307 30699 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:39.527396 30699 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:39.527752 30699 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:59:39.528514 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:39.529042 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:39.529116 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:39.529320 30699 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:59:39.533823 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.536881 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.542948 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3276,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":246}
I20240501 19:59:39.543718 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3): 4834 bytes on disk
I20240501 19:59:39.544064 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 19:59:39.547073 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:39.560465 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(a170766f497f491ba9b45f92f2a0e751) complete. Timing: real 0.042s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1655,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 19:59:39.561256 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(a170766f497f491ba9b45f92f2a0e751): 4581 bytes on disk
I20240501 19:59:39.561676 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(a170766f497f491ba9b45f92f2a0e751) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 19:59:39.605487 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:39.610975 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:39.636220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.636618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.639345 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.641847 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.642923 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.645906 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.733479 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.734192 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.734970 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.740247 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.750556 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.759172 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.855683 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.858006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.860893 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.862090 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.862723 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): perf score=0.033340
I20240501 19:59:39.877864 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.884158 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5771,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":116,"threads_started":2,"wal-append.queue_time_us":3100}
I20240501 19:59:39.885164 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): 4834 bytes on disk
I20240501 19:59:39.885650 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 19:59:39.893086 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:39.896457 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:39.898895 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:39.899616 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.913698 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(26b12620ad9a404d9ab70e8506ad1a39): perf score=0.033354
I20240501 19:59:39.922201 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.924870 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:39.927587 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:39.950177 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(26b12620ad9a404d9ab70e8506ad1a39) complete. Timing: real 0.036s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4357,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":12904,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109,"threads_started":2,"wal-append.queue_time_us":13182}
I20240501 19:59:39.950861 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39): 4580 bytes on disk
I20240501 19:59:39.951177 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:59:39.962203 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:39.998728 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.012828 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:40.020640 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:40.021515 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.021754 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.027319 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.035915 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.046066 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): perf score=0.033393
W20240501 19:59:40.047806 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:40.050406 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.064239 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(d5faf889870246b78b82e19c041dc119): perf score=0.033400
I20240501 19:59:40.068449 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.022s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1706,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1208,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 19:59:40.069284 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): 4834 bytes on disk
I20240501 19:59:40.069792 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240501 19:59:40.075786 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.080485 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(d5faf889870246b78b82e19c041dc119) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":113,"threads_started":2,"wal-append.queue_time_us":130}
I20240501 19:59:40.081537 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(d5faf889870246b78b82e19c041dc119): 4921 bytes on disk
I20240501 19:59:40.081933 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(d5faf889870246b78b82e19c041dc119) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.135412 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.136231 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.136566 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.139587 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.149330 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.154085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:40.168804 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:40.177788 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:40.215224 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 19:59:40.249080 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.257128 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.257505 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.269096 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.269369 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.270177 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.279584 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.280889 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.282303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.290701 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:40.302240 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.303716 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.304478 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:40.376740 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:40.379323 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:40.387742 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(5d032fb77ac944629a45b808cc76c2f1): perf score=0.033343
I20240501 19:59:40.405607 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.407696 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.408250 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.408735 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(5d032fb77ac944629a45b808cc76c2f1) complete. Timing: real 0.021s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7640,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":2842,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":333}
I20240501 19:59:40.410178 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67): perf score=0.033348
I20240501 19:59:40.410553 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.413877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.425642 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:40.430017 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.438338 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.027s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":2034,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":475,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":53,"threads_started":1,"wal-append.queue_time_us":3256}
I20240501 19:59:40.439008 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67): 4834 bytes on disk
W20240501 19:59:40.440450 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:40.439354 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.445549 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(5d032fb77ac944629a45b808cc76c2f1): 4581 bytes on disk
I20240501 19:59:40.446044 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(5d032fb77ac944629a45b808cc76c2f1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.453403 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(219742bd987e4a1990233fa8c972c232): perf score=0.033365
I20240501 19:59:40.464601 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1085,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":359,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":114,"threads_started":2,"wal-append.queue_time_us":115}
I20240501 19:59:40.465256 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232): 4921 bytes on disk
I20240501 19:59:40.465554 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 19:59:40.519707 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 19:59:40.572394 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(219742bd987e4a1990233fa8c972c232): perf score=0.033398
W20240501 19:59:40.578738 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:40.583819 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67): perf score=0.033399
I20240501 19:59:40.587507 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":244,"threads_started":2,"wal-append.queue_time_us":117}
I20240501 19:59:40.588343 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232): 4921 bytes on disk
I20240501 19:59:40.588712 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.589525 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(5d032fb77ac944629a45b808cc76c2f1): perf score=0.033400
W20240501 19:59:40.614207 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.615470 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(5d032fb77ac944629a45b808cc76c2f1) complete. Timing: real 0.026s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1134,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5270,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":2354}
I20240501 19:59:40.616211 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(5d032fb77ac944629a45b808cc76c2f1): 4581 bytes on disk
I20240501 19:59:40.616602 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(5d032fb77ac944629a45b808cc76c2f1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.618942 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.035s user 0.011s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":663,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":5727,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3386,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":10496,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":6689}
I20240501 19:59:40.622267 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67): 4834 bytes on disk
I20240501 19:59:40.623070 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.624744 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.628953 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.628995 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.635475 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.640223 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.649547 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:40.684087 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:40.691183 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:59:40.696049 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.697275 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(318eb8fd9972435a8e51f63bac077f09): perf score=0.033335
I20240501 19:59:40.717399 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(318eb8fd9972435a8e51f63bac077f09) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1638,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":130}
I20240501 19:59:40.718112 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09): 4579 bytes on disk
I20240501 19:59:40.718160 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff): perf score=0.033341
I20240501 19:59:40.718777 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.722940 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.726686 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.729178 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.733489 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.734473 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.735211 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1745,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":170}
I20240501 19:59:40.735954 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(269047cc3a504d7488747166efce71fc): perf score=0.033344
W20240501 19:59:40.745129 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:40.752761 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.764618 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":1298,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":5206}
I20240501 19:59:40.765357 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff): 4921 bytes on disk
I20240501 19:59:40.765705 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.766330 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc): 4834 bytes on disk
I20240501 19:59:40.766650 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240501 19:59:40.798272 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:40.831180 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.832650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.838234 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.840739 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.844836 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:40.846828 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 19:59:40.847903 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:40.856976 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:40.867847 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff): perf score=0.033380
I20240501 19:59:40.874481 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(269047cc3a504d7488747166efce71fc): perf score=0.033378
I20240501 19:59:40.885324 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.017s user 0.014s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1335,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":51,"threads_started":1,"wal-append.queue_time_us":115}
W20240501 19:59:40.885805 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:40.886145 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff): 4921 bytes on disk
I20240501 19:59:40.886516 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.890586 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.016s user 0.011s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1200,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":188}
I20240501 19:59:40.891217 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc): 4834 bytes on disk
I20240501 19:59:40.891578 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:59:40.955348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.962271 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.968288 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:40.975206 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.975381 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:40.978129 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:40.986104 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:40.988363 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:40.998466 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.008603 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.021607 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.023842 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:41.036495 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:41.090690 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:41.115559 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:41.122884 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.124478 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.125432 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.134388 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.134760 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.149140 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:41.220155 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:41.221841 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(54d2db0f85ac4a90a1df779f89398463): perf score=0.033344
I20240501 19:59:41.223611 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.226456 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.229269 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.233824 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:41.246539 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 19:59:41.249120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.250593 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10814,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":881024,"thread_start_us":107,"threads_started":2,"wal-append.queue_time_us":7850}
I20240501 19:59:41.251355 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463): 4834 bytes on disk
I20240501 19:59:41.251672 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.255874 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.268455 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(54d2db0f85ac4a90a1df779f89398463): perf score=0.033356
I20240501 19:59:41.294517 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.026s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3378,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":2,"wal-append.queue_time_us":2147}
I20240501 19:59:41.295359 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463): 4834 bytes on disk
I20240501 19:59:41.295760 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.335096 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.337031 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.340950 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.342258 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:41.367165 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:41.369375 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.372320 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.393426 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026): perf score=0.033395
I20240501 19:59:41.394122 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(25ea8d7ab1834abfa34ff2357aa554b0): perf score=0.033394
I20240501 19:59:41.409574 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.016s user 0.011s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":135,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1687,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":115,"threads_started":2,"wal-append.queue_time_us":1059}
I20240501 19:59:41.410490 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 19:59:41.411446 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.414027 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.417884 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.420002 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.422116 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.427119 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(25ea8d7ab1834abfa34ff2357aa554b0) complete. Timing: real 0.033s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":878,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":8258,"drs_written":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":294,"threads_started":2,"wal-append.queue_time_us":4855}
I20240501 19:59:41.427911 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0): 4921 bytes on disk
I20240501 19:59:41.428251 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.429225 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026): perf score=0.033401
I20240501 19:59:41.429651 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.440918 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:41.442754 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:59:41.442860 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:41.464466 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.035s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1321,"drs_written":1,"lbm_read_time_us":275,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":19732}
I20240501 19:59:41.465356 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 19:59:41.465855 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.524868 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.526279 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.528187 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.529940 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.531383 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:41.535558 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:41.541366 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:41.562844 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:41.571508 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:41.632321 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.632361 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.641048 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.641883 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.652256 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.657068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:41.661859 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:59:41.672179 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 19:59:41.720952 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.721179 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.722451 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.730037 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.733254 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.750170 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.754065 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(2d6621e1ff294dc9a474e076873cc72c): perf score=0.033370
I20240501 19:59:41.780862 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(2d6621e1ff294dc9a474e076873cc72c) complete. Timing: real 0.027s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9530,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":2437,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":122,"threads_started":2,"wal-append.queue_time_us":3463}
I20240501 19:59:41.781579 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(2d6621e1ff294dc9a474e076873cc72c): 4834 bytes on disk
I20240501 19:59:41.782357 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(2d6621e1ff294dc9a474e076873cc72c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:59:41.793547 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.794063 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.797861 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): perf score=0.033385
I20240501 19:59:41.801584 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.802553 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.803015 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.807255 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.821424 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.023s user 0.013s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3306,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":114,"threads_started":2,"wal-append.queue_time_us":7425}
I20240501 19:59:41.822187 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): 4921 bytes on disk
I20240501 19:59:41.822484 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 19:59:41.837699 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:41.908780 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:41.912724 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:41.914101 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:41.916855 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:41.921008 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:41.929845 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:41.970831 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.007602 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.013115 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.018903 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.019030 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.042135 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.044381 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.096705 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.099118 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.100963 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.113843 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.127904 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.130012 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:42.164017 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:42.168308 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.169879 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.182193 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.182621 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.183807 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.200024 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.219250 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d): perf score=0.033378
I20240501 19:59:42.230866 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":144,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1418,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":111}
I20240501 19:59:42.231467 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d): 4834 bytes on disk
I20240501 19:59:42.231761 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:59:42.286964 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1): perf score=0.033396
I20240501 19:59:42.297825 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1359,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":354,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":107,"threads_started":2,"wal-append.queue_time_us":134}
I20240501 19:59:42.298497 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1): 4581 bytes on disk
W20240501 19:59:42.298570 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:59:42.298828 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 19:59:42.378476 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:59:42.423498 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:42.434604 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.438194 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.438601 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.440093 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.441087 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.458634 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.517671 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.518850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.519932 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.520506 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.529948 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.542349 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:42.565621 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.619725 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.626464 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.626734 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.627478 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:42.638392 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:42.638825 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.639132 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:42.685595 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:59:42.718017 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.728472 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:42.729616 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.732705 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.737423 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.741075 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.741676 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.749583 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:42.768203 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:42.793740 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:42.795076 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.804874 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.806988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.807368 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.809244 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.814996 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.817224 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:42.840142 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:42.883026 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.888178 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.891355 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:42.893543 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.893823 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:42.905737 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.913700 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:42.945355 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:42.981182 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:42.983084 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:42.986912 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:42.987982 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:42.999660 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.004874 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.066735 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:43.097931 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.099781 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.101480 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:43.102295 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:43.103994 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.105358 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.116003 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.170658 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:43.184238 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.184551 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.194692 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:43.195895 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.198189 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.213351 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.254971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.259721 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.259753 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.261194 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.273736 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:43.279328 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:43.286120 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:43.289726 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:43.366675 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.367753 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.368129 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.371788 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.372256 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:43.385499 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:43.394946 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.419277 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:59:43.454469 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:43.461124 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:43.477968 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.479993 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.483494 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.485062 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.487675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.507400 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.544551 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:43.546725 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.549021 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.551056 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.553383 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.562252 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.569496 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:43.631354 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.634894 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.636142 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:43.637097 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.642459 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.659480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:43.709578 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:43.730159 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.732023 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.732088 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.736675 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.737754 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.741050 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.760152 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:43.794874 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:43.829317 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:43.832356 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.837703 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.840521 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.848145 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.857666 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.864960 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:59:43.873833 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.904863 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:43.905555 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:43.908007 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:43.908365 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.912583 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:43.921111 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:43.973893 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:43.992261 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:44.112973 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:44.157100 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:44.160228 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.165702 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.165783 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.174470 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.176167 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.185472 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:44.212589 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:44.268879 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.275135 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.278514 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.284602 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.284662 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.295085 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.304124 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.304478 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.320447 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.321708 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.337675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.347615 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.429358 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.429976 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.432453 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.433823 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.438530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.441723 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.530344 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.530493 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.530498 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.531255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.532392 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.544503 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:44.568373 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:44.611649 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:44.642652 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.651064 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.651305 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.654716 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.661998 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.681442 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.707945 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.708032 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.719379 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.721974 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.729666 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.735201 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:44.794000 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:44.803841 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.814973 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.815538 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.817247 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.818229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.829532 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.914840 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:44.918062 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:44.920473 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:44.924769 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:44.926129 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:44.938556 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:44.976644 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:45.032449 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.037154 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.038355 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.038657 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.069594 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.077755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.081564 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.084650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.088616 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.097697 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.111549 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:45.118664 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.121181 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:45.125241 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.195418 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.196624 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.200978 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.202005 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.202677 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:45.204531 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:45.207387 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:45.221879 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 19:59:45.224162 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.249017 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.265770 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.298657 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:45.303843 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.304373 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.307365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.308353 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.309477 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.314101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:45.352099 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.369119 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.372263 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:59:45.389366 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:45.411060 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.416206 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.416477 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.426172 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.427343 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.428400 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.468279 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.468644 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.475035 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.475845 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.476883 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.481410 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.551836 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.556660 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.557314 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.563479 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:45.569614 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 19:59:45.570096 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.571174 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:45.585134 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:45.605867 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.644021 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:59:45.704290 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:45.773277 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.773837 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.775337 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:45.782100 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.783627 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.783891 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.858307 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(95b8c1a34cc94d29986289939b2159bb): perf score=0.033344
I20240501 19:59:45.876148 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:45.878319 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:45.885897 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:45.887135 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:45.898821 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(95b8c1a34cc94d29986289939b2159bb) complete. Timing: real 0.040s user 0.025s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":161,"dirs.queue_time_us":6542,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":238,"threads_started":3,"wal-append.queue_time_us":269}
I20240501 19:59:45.900300 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3): perf score=0.033354
I20240501 19:59:45.913789 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:45.915274 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:45.915787 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.916631 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:45.922451 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.022s user 0.017s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":138}
I20240501 19:59:45.923166 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3): 4921 bytes on disk
I20240501 19:59:45.923622 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 19:59:45.924208 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb): 4581 bytes on disk
I20240501 19:59:45.924475 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
W20240501 19:59:45.939934 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 19:59:45.940712 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:45.942152 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1): perf score=0.033357
I20240501 19:59:45.970697 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.028s user 0.026s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":8659,"dirs.queue_time_us":6824,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":868,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":193,"threads_started":3,"wal-append.queue_time_us":118}
I20240501 19:59:45.972144 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1): 4581 bytes on disk
I20240501 19:59:45.972543 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:59:45.973218 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d): perf score=0.033365
W20240501 19:59:45.981581 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:45.991744 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.018s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1084,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":55,"threads_started":1,"wal-append.queue_time_us":125}
I20240501 19:59:45.992409 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d): 4834 bytes on disk
I20240501 19:59:45.992692 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 19:59:46.023339 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:46.077157 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:46.078729 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:59:46.081142 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.099438 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.101296 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.101688 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.102334 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:46.174422 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:59:46.175287 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:46.176069 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(318eb8fd9972435a8e51f63bac077f09): perf score=0.033381
W20240501 19:59:46.190249 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:59:46.193616 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:46.208819 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(318eb8fd9972435a8e51f63bac077f09) complete. Timing: real 0.033s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":85,"threads_started":1,"wal-append.queue_time_us":10800}
I20240501 19:59:46.210616 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff): perf score=0.033378
I20240501 19:59:46.215973 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.228665 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.233450 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.237792 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.241539 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.241593 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.244319 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(6fd958b083bb430a99130a18f91acdf7): perf score=0.033383
I20240501 19:59:46.279093 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.068s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1376,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":2127}
I20240501 19:59:46.280297 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff): 4921 bytes on disk
I20240501 19:59:46.280771 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(c3ee13ec9bb1468f90cfc184455f9fff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.281042 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(6fd958b083bb430a99130a18f91acdf7) complete. Timing: real 0.037s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":22020,"compiler_manager_pool.run_cpu_time_us":14,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":350,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":560}
I20240501 19:59:46.281455 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09): 4579 bytes on disk
I20240501 19:59:46.281797 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.281873 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(6fd958b083bb430a99130a18f91acdf7): 4581 bytes on disk
I20240501 19:59:46.282212 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(6fd958b083bb430a99130a18f91acdf7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.282502 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67): perf score=0.033385
I20240501 19:59:46.283438 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(a170766f497f491ba9b45f92f2a0e751): perf score=0.033343
W20240501 19:59:46.290269 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 19:59:46.298653 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(a170766f497f491ba9b45f92f2a0e751) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1059,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":159}
I20240501 19:59:46.299623 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(a170766f497f491ba9b45f92f2a0e751): 4581 bytes on disk
I20240501 19:59:46.299983 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(a170766f497f491ba9b45f92f2a0e751) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.313768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.314502 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.024s user 0.012s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":246,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":1249,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":85,"threads_started":1,"wal-append.queue_time_us":730}
I20240501 19:59:46.315768 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67): 4834 bytes on disk
I20240501 19:59:46.316840 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.317250 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(8ddd522180bd4e53906cd9572f82ab67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.319123 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.321002 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.321303 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:46.322047 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 19:59:46.335947 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.418865 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:46.423599 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:46.424054 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.425045 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.425809 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.432719 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.447017 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:46.467099 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:46.520017 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.521068 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.529345 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.537510 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.538750 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.541963 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.551541 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6): perf score=0.033394
I20240501 19:59:46.566987 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1273,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":273}
I20240501 19:59:46.567739 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6): 4834 bytes on disk
I20240501 19:59:46.568013 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.620026 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.628160 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.629578 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.639611 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.656894 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:46.671792 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:46.673573 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.697062 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.697121 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.697610 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.704571 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.711594 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:46.713235 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:46.722033 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:46.770403 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:46.806241 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.806288 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.807145 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.809835 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.819386 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(19f79cd7d4ec4b4697a2637e6d14a892): perf score=0.033401
I20240501 19:59:46.819427 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(2bc8fbbdc94e49d2b8b639083e083df3): perf score=0.033363
I20240501 19:59:46.824944 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:46.825188 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:46.826004 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.848834 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(2bc8fbbdc94e49d2b8b639083e083df3) complete. Timing: real 0.028s user 0.011s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":2323,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":11387}
I20240501 19:59:46.849874 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3): 4921 bytes on disk
I20240501 19:59:46.850343 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 19:59:46.856391 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(19f79cd7d4ec4b4697a2637e6d14a892) complete. Timing: real 0.037s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1113,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":578,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":179}
I20240501 19:59:46.857280 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892): 4921 bytes on disk
I20240501 19:59:46.857758 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
W20240501 19:59:46.876689 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:46.930824 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:46.931298 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:46.937062 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:46.944402 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:46.962617 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:46.970849 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.034842 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.036726 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.047264 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.061239 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.065086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.070102 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.101996 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8): perf score=0.033352
I20240501 19:59:47.109290 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(19d8d7d255494a49920849d10b800fa7): perf score=0.033362
I20240501 19:59:47.116546 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.014s user 0.008s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1392,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":133}
I20240501 19:59:47.117331 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8): 4581 bytes on disk
I20240501 19:59:47.117676 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:59:47.123701 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":643}
I20240501 19:59:47.124576 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7): 4921 bytes on disk
I20240501 19:59:47.125211 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 19:59:47.139614 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.141816 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.145144 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.145748 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.146066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:47.165290 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.170993 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:47.235656 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:47.249382 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.249867 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.253506 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.254297 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:47.268790 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:47.269804 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.273550 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.368134 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.369246 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(d5faf889870246b78b82e19c041dc119): perf score=0.033357
I20240501 19:59:47.376626 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.388398 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.390136 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.394080 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(d5faf889870246b78b82e19c041dc119) complete. Timing: real 0.025s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1448,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":55,"threads_started":1,"wal-append.queue_time_us":116}
I20240501 19:59:47.394933 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(d5faf889870246b78b82e19c041dc119): 4921 bytes on disk
I20240501 19:59:47.395315 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(d5faf889870246b78b82e19c041dc119) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 19:59:47.403188 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.403748 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.418026 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.418444 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.418960 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.424445 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.425153 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:47.433708 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 19:59:47.435678 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:47.450384 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:59:47.501010 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.528497 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.530689 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.530925 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.534072 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.541074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.542613 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:47.560705 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:59:47.582173 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:47.627013 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(54d2db0f85ac4a90a1df779f89398463): perf score=0.033367
W20240501 19:59:47.633409 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.639001 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":384,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":114,"threads_started":2,"wal-append.queue_time_us":106}
I20240501 19:59:47.639657 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463): 4834 bytes on disk
I20240501 19:59:47.639930 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(54d2db0f85ac4a90a1df779f89398463) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:59:47.646665 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3): perf score=0.033383
I20240501 19:59:47.658064 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.011s user 0.009s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1255,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":356,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":153}
I20240501 19:59:47.658689 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3): 4834 bytes on disk
I20240501 19:59:47.658964 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(b4f0699b5bdf44499c87ff1541cd2ad3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
W20240501 19:59:47.715631 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.728528 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.729897 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.730401 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.731945 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.745777 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.753058 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:47.756515 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:47.771421 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.839531 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.845161 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.845582 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.846808 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:47.848213 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.850258 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:47.854666 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:47.879217 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.890950 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): perf score=0.033337
W20240501 19:59:47.900364 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:47.904171 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1359,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":55,"threads_started":1,"wal-append.queue_time_us":124}
I20240501 19:59:47.905001 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea): 4834 bytes on disk
I20240501 19:59:47.905367 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(5eb1f0dcd4cf49cd969eb28f1eb5d9ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 19:59:47.906208 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:47.941999 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:47.946926 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:47.948840 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.949862 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:47.956185 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:47.962324 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:48.020195 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.067925 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.068783 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.071768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.072611 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.080019 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:48.081051 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.088599 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.090292 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.094349 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.118993 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.119714 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.134112 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.141048 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:48.185437 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:48.217206 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.222821 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:48.224427 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.225746 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.226442 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.230042 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.231110 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:48.233542 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.245816 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:48.296602 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:48.320029 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.325413 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.325688 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.326740 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.327580 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:48.335420 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.343600 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.358850 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:48.428999 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:48.441201 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.441439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.445884 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.452490 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.452658 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.454754 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.496235 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.501976 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.504557 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.507215 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.514173 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.517901 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:48.603155 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:59:48.605762 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.613435 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.620880 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.621146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.623430 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.633574 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:48.663862 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:48.688757 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.728027 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.728595 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.734650 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.735239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.740618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.745880 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:48.793673 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.800244 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.800292 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.805056 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.805948 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:48.814157 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.815622 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:48.849511 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:48.863977 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.899319 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:48.901420 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:48.905197 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:48.907986 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:48.909058 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:48.912603 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:48.925573 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.010588 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.014174 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.022343 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.023018 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.024504 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.033361 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.135871 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.137378 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.138084 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.141327 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.145175 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.146869 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.151254 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.161854 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.171876 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.180939 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.183916 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.186321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:49.234719 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:49.268028 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:49.292387 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.293035 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.295044 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.298048 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.309195 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.317031 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:49.331631 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:49.421952 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:49.482049 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.483351 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.486079 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.486851 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.488327 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.507679 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.604120 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.607703 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.614487 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.618944 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.622038 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.629539 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:49.700497 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:49.709427 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.710168 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.712690 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.713141 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.713505 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.714051 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:49.803714 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:49.818243 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.821635 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.824712 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.834380 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.835283 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.836800 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:49.844288 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:49.853003 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:49.863233 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:49.865206 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:49.870175 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:49.871887 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.874570 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:49.877434 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:49.883991 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:49.960059 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:49.967468 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:49.978605 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:49.978683 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:49.980255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.981017 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:49.981110 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.003330 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.082171 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.083588 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.084372 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:50.085153 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:50.090795 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.096560 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.099705 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.194247 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.197939 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.200103 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.207676 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.211695 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.221982 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:50.222337 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 75 similar messages]
I20240501 19:59:50.232934 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.236106 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:50.237123 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.241325 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.248739 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.256841 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.257488 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:50.281690 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.353242 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.359951 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.366106 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.372772 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.374691 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.376180 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:50.444558 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:50.454805 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:50.466509 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.467646 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.469287 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.474593 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.482743 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.491922 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:50.514945 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.570078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:50.578018 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 19:59:50.579016 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:50.581290 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:50.581709 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.586586 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:50.590828 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.596308 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.597926 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.600306 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.609823 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.620429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.633394 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.633780 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.642441 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.649734 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.669664 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(26b12620ad9a404d9ab70e8506ad1a39): perf score=0.033353
I20240501 19:59:50.684876 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(26b12620ad9a404d9ab70e8506ad1a39) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1756,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1056,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":105,"threads_started":2,"wal-append.queue_time_us":112}
I20240501 19:59:50.685770 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39): 4580 bytes on disk
W20240501 19:59:50.685961 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.686167 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 19:59:50.736032 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:50.736372 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:50.739111 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.743907 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.746110 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.748450 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.755337 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.756119 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:50.786975 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:50.797161 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:50.840693 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.842780 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.844116 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.853317 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.853571 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.856935 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:50.929901 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:50.941071 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 19:59:50.951336 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:50.958880 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:50.960837 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:50.963184 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:50.969046 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:50.978888 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.002576 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.007459 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.007519 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.008761 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.020689 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:51.023415 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:51.024477 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:51.052454 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:59:51.059852 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:51.247702 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:59:51.276167 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.283464 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.286962 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.292770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.298627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.299283 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:51.313694 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:51.321117 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:51.369693 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.370306 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.375535 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.377264 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:51.383360 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 19:59:51.384649 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.385794 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:51.409946 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:51.463394 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.466229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:51.467339 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 19:59:51.467581 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.471693 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.475566 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.476666 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:51.557415 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:51.571468 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:51.574503 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:51.575759 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:51.577813 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:51.579042 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.579490 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.582840 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.600528 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.647532 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.648159 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.655586 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.656816 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.658473 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.669188 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.723917 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.725379 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.727358 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.728283 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.728803 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.745432 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:51.805953 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:51.818845 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.819373 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.821180 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:51.822919 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.830562 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.849062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.927129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:51.929468 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:51.949185 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.950475 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:51.951093 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:51.959471 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.004062 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.005558 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.009079 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.009151 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.016458 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.023901 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:52.057394 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:52.075014 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:52.100889 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.107875 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.108443 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.109326 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.118166 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.122093 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:52.163511 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:52.209918 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.210175 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.219569 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.227185 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.233403 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.237653 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:52.239820 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.301604 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:52.321848 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.323652 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.329439 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.336391 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.337630 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:52.337637 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:52.365589 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.374634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.377180 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.387746 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.388206 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.389145 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:52.399503 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:52.401631 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.481040 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.481698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.485558 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.487746 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.490350 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.506033 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.576383 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.576839 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.581717 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.585083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.589028 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.593139 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:52.601334 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:52.604140 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.621866 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:52.629643 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:52.678356 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.698971 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:52.699985 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.705372 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.707890 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:52.711432 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.712002 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.714066 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.720530 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:52.727208 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:52.727283 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.729148 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:52.737210 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:52.773622 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:52.883011 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.940404 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.966821 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:59:52.994130 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:53.030484 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:53.054358 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.055178 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:53.068697 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:53.070878 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.089666 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.090554 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.092581 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:53.094035 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:53.094308 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:53.099835 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.101068 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.117158 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:53.118513 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.119477 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.121676 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.130138 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.221603 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.223074 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.226218 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.227057 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.237924 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.241760 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:53.267271 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:53.286041 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.334179 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.336035 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.336411 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.345207 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.345456 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.346451 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:53.356253 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.432507 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:53.436048 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:53.436230 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.436609 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.441325 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.443845 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:53.455439 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:53.459791 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.493399 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.503050 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.504496 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.506080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.509073 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.514679 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.598676 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.599584 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.601953 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:53.602872 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.605440 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.607015 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:53.621980 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:53.622431 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:53.681735 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:53.705111 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.706982 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.707115 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.711815 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.712685 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.719615 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:53.789130 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:53.805709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.807895 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:53.810801 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.827670 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.837357 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.841974 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:53.856884 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:59:53.858500 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:53.865298 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:53.865383 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:53.870340 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:53.910491 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:53.971733 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:53.971901 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.053133 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.055814 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.056975 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.058904 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:54.066676 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:54.068209 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.074600 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.157982 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.163098 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.166078 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.167975 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.178284 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:54.196017 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.215898 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.216534 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.221287 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:54.221531 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:54.224247 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.225945 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.227305 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.235767 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.274793 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1): perf score=0.033377
I20240501 19:59:54.289494 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.015s user 0.009s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1233,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":109}
I20240501 19:59:54.290479 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1): 4581 bytes on disk
I20240501 19:59:54.290818 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(156d2ba7ff2c403db2f63cbae4acfec1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:59:54.327872 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.332948 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.333411 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.339424 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.350404 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.356719 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.438144 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.438500 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.439342 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.441004 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.453936 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.458673 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:54.540331 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.540449 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.541743 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.548619 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.553901 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.557498 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.562884 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.597100 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.601982 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.611883 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.617056 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:54.617782 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.617828 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.625144 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:54.651532 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:54.702977 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248): perf score=0.033395
I20240501 19:59:54.722572 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.019s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1178,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":415,"threads_started":2,"wal-append.queue_time_us":141}
I20240501 19:59:54.723305 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248): 4834 bytes on disk
I20240501 19:59:54.723713 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 19:59:54.765434 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:54.784859 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:54.791321 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.875635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.876662 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.879489 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.881789 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:54.895092 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.897475 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.902916 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:54.937464 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:54.951485 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:54.953375 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:54.957270 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:54.958106 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:54.967170 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:54.969197 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.053485 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.054273 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.056476 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.059239 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.062039 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.075559 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:55.124560 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:55.158393 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.159584 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.160038 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.163940 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.164723 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:55.173521 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:55.176539 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.189442 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:55.203730 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.243165 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.243970 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.247829 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:55.252699 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 19:59:55.253402 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:55.259426 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.262164 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.267657 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:55.284369 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:55.286850 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:55.309809 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.318868 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.322064 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.322753 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.322880 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.326227 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:55.339254 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:55.339835 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.422750 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.424652 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.425002 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.425854 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.428206 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.430505 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.532848 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.534803 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.539561 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:55.542177 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:55.544433 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.549422 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.550050 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.625870 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.627669 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.632438 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.633997 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.642647 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.650288 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:55.678432 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 19:59:55.681242 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:55.695863 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.696348 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.701169 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.710142 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.721299 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:55.722266 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:55.728783 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:55.742511 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:55.798759 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:55.799857 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.801362 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.808773 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.812960 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.821178 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.822463 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:55.882074 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.905911 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:55.911259 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.911937 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:55.914119 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:55.918104 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:55.918241 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:55.922447 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:55.928862 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:55.956810 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:55.957788 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:59:55.992282 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:55.992302 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.004135 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.004786 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.008121 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.010639 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:56.013079 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 19:59:56.013780 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:56.072242 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.076790 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.077371 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.079510 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:56.086432 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:56.090231 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.090593 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.177947 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): perf score=0.033369
I20240501 19:59:56.183107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.186111 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.190613 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.192131 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.198685 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.202270 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.024s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":485,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":116,"threads_started":2,"wal-append.queue_time_us":107}
I20240501 19:59:56.203594 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): 4921 bytes on disk
I20240501 19:59:56.207182 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 19:59:56.218727 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:56.224202 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:59:56.296097 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:56.304132 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.304649 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.305110 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.310724 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.311764 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.323868 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:56.334307 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:56.341787 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.362103 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.362970 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.370754 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.379588 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.383502 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:56.454669 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 19:59:56.464061 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3): perf score=0.033401
I20240501 19:59:56.474905 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1333,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":375,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":127}
I20240501 19:59:56.475526 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3): 4921 bytes on disk
I20240501 19:59:56.475816 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(0a1bbe74dbc74fa687b0a64d3aa203f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240501 19:59:56.477506 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:59:56.494151 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:56.503238 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:56.663725 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.667215 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.670212 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.670694 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.678149 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.687885 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.704185 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248): perf score=0.033334
I20240501 19:59:56.728024 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.728749 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.732203 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.733754 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: FlushMRSOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.029s user 0.014s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9632,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":107,"threads_started":2,"wal-append.queue_time_us":106}
I20240501 19:59:56.734519 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248): 4834 bytes on disk
I20240501 19:59:56.734846 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(19bfb955b9c94fdb8e0381e5b1855248) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240501 19:59:56.745114 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:56.757508 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:56.765810 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.766806 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.806546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.808977 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.811478 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.821194 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.823055 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:56.841481 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:56.856593 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:56.859503 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:56.924350 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:56.924978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:56.925439 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:56.929654 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.930238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:56.933048 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:56.933246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:56.964457 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.029183 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.043782 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.044734 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.048102 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.055974 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.063091 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.104753 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.113577 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.116992 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.117597 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.119199 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.119918 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:57.169612 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.197657 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.201159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.202580 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.203243 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.205862 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.212975 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:57.226764 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:57.231775 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(25ea8d7ab1834abfa34ff2357aa554b0): perf score=0.033347
I20240501 19:59:57.243932 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(25ea8d7ab1834abfa34ff2357aa554b0) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":110,"threads_started":2,"wal-append.queue_time_us":100}
I20240501 19:59:57.244575 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0): 4921 bytes on disk
I20240501 19:59:57.244899 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 19:59:57.291106 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:57.300041 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:57.315490 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.320248 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.320343 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.321739 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.322175 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.331794 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.341199 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6): perf score=0.033390
W20240501 19:59:57.355659 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.376399 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: FlushMRSOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":3110}
I20240501 19:59:57.377242 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6): 4834 bytes on disk
I20240501 19:59:57.377600 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(9fcd6776aebf47c99f05850de83b36e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 19:59:57.424044 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.444802 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.449656 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.453672 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.456002 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.458782 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.470698 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.472806 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.473050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.491189 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.496785 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.497824 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026): perf score=0.033348
I20240501 19:59:57.507382 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:57.513988 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.524401 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.533329 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.032s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1215,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":166}
I20240501 19:59:57.534281 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 19:59:57.534744 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 19:59:57.597846 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.604220 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.612529 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.613353 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.614950 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.630910 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:57.638438 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:59:57.669073 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:57.671303 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.708876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.714995 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.715382 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.721858 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:57.722330 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:57.724186 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:57.727296 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:57.730952 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(219742bd987e4a1990233fa8c972c232): perf score=0.033386
I20240501 19:59:57.739219 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.756487 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.025s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6702,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":137}
I20240501 19:59:57.757359 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232): 4921 bytes on disk
I20240501 19:59:57.757660 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(219742bd987e4a1990233fa8c972c232) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 19:59:57.806622 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:57.814781 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.827970 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:57.832849 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.833253 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.838277 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.851657 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.865291 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(19d8d7d255494a49920849d10b800fa7): perf score=0.033398
I20240501 19:59:57.871524 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.883134 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:57.885107 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:57.887632 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.890969 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.893069 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.904690 7046 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 19:59:57.904831 7046 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 19:59:57.906386 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:57.906548 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:59:57.907107 7046 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 19:59:57.907194 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:59:57.907548 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:57.907629 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:57.907703 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 19:59:57.907841 7046 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:59:57.922863 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.052s user 0.015s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":137,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":7614}
I20240501 19:59:57.923776 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7): 4921 bytes on disk
I20240501 19:59:57.924151 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(19d8d7d255494a49920849d10b800fa7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 19:59:57.986205 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:57.990952 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:57.991199 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:57.995895 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:57.997815 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.006814 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.009016 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling FlushMRSOp(269047cc3a504d7488747166efce71fc): perf score=0.033363
I20240501 19:59:58.033686 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: FlushMRSOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.025s user 0.009s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":629,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1234,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":142}
I20240501 19:59:58.034538 13084 maintenance_manager.cc:419] P e3c6b84864cb4358a9d7e0f2a65ba83d: Scheduling UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc): 4834 bytes on disk
I20240501 19:59:58.034945 12974 maintenance_manager.cc:643] P e3c6b84864cb4358a9d7e0f2a65ba83d: UndoDeltaBlockGCOp(269047cc3a504d7488747166efce71fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 19:59:58.102936 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.108036 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.109933 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.117452 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.119045 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.147845 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:58.161159 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:58.175503 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(2bc8fbbdc94e49d2b8b639083e083df3): perf score=0.033335
I20240501 19:59:58.193514 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(2bc8fbbdc94e49d2b8b639083e083df3) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":306}
I20240501 19:59:58.194432 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3): 4921 bytes on disk
I20240501 19:59:58.194826 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 19:59:58.209376 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.210820 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:58.211426 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.212939 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.215485 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.232338 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:58.234320 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.237607 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:58.257941 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 19:59:58.276226 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:58.287423 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d): perf score=0.033368
I20240501 19:59:58.298461 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: FlushMRSOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":1258,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":414,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":118,"threads_started":2,"wal-append.queue_time_us":118}
I20240501 19:59:58.299120 12820 maintenance_manager.cc:419] P 14002a0a4c674b438c78ca5db73637cc: Scheduling UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d): 4834 bytes on disk
I20240501 19:59:58.299408 12710 maintenance_manager.cc:643] P 14002a0a4c674b438c78ca5db73637cc: UndoDeltaBlockGCOp(2def1cda07c349c69b3abe25804d3d1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 19:59:58.306969 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:59:58.311399 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 19:59:58.346900 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:58.431092 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.433053 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.438315 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.439163 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.440196 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.454916 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:58.461338 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.537875 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.542912 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.545821 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.555686 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.558359 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.567031 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.584800 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.589075 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:58.591380 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.602970 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.614722 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.627197 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.627975 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.702507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.706686 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.706987 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.709684 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.710031 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:58.716732 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.731801 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:58.805862 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:58.810276 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.810740 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:58.812856 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:59:58.821738 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.823143 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.824576 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:58.833041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:58.897298 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:58.911643 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.914647 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.921382 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.926769 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.938206 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:58.940109 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:58.940413 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.948570 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:58.952700 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:58.953310 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:58.964561 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.966415 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:58.970185 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:59.000177 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.058727 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): perf score=0.033348
W20240501 19:59:59.066145 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.084817 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.089740 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.090073 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.091820 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.093278 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.097182 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.121626 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: FlushMRSOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.063s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":123,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1570,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":4730}
I20240501 19:59:59.122519 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16): 4921 bytes on disk
I20240501 19:59:59.122912 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(dbc6ed9f4bf34b1caa462aa9d85c7a16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 19:59:59.189754 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.191066 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.191951 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.197765 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.201445 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.210304 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.303608 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.303608 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.307438 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.308372 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 19:59:59.308943 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.310547 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.310911 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.371768 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.372689 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.373085 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.374711 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.376153 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.384423 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 19:59:59.422628 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:59.449556 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8): perf score=0.033362
I20240501 19:59:59.465528 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: FlushMRSOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":3183,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":186}
I20240501 19:59:59.466794 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8): 4581 bytes on disk
I20240501 19:59:59.467339 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(6edb640b8cc04d20b0815b1beb45d9c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 19:59:59.468742 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:59:59.469194 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 19:59:59.479082 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.479357 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.484236 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.484772 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.496918 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.497177 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.578281 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.578747 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.580427 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.592211 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.600651 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.602205 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:59.637498 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.669597 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.673306 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.674818 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.679157 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.684634 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.704891 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.732211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.736665 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.737251 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.744838 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.750443 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 19:59:59.755507 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.757607 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 19:59:59.826030 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 19:59:59.838490 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.839141 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.840426 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.849166 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.858748 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.869388 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 19:59:59.914628 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:59:59.987370 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:59:59.988219 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:59:59.990628 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:59:59.994804 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:59:59.997074 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:59:59.998579 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:00.000599 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:00.055320 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:00.099598 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:00:00.101912 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:00.146461 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.152150 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.152624 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.156364 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.169039 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:00.171245 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:00.236250 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:00.248864 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.252411 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.256165 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.258672 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.260246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:00.264101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:00.293216 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 20:00:00.293838 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:00:00.303303 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:00:00.322618 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:00.338696 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.342586 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.345901 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:00.349110 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.353248 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.378593 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.419576 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.420203 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.421295 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.422349 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.422616 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.425494 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:00.452340 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:00.519701 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.520843 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.522043 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.523337 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:00.533022 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.541939 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:00.553263 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:00:00.601986 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:00.602770 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:00.612776 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:00.627878 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.630034 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.630299 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.637173 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.648618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.661392 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:00.699209 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:00:00.702848 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:00.709988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.710006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.713146 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.721400 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.729935 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:00.739202 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:00.750137 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.773782 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:00.778764 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.780423 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.788409 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.797470 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.799659 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:00.822059 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:00.830823 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:00.889554 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:00.892122 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:00.893337 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:00.897262 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.903002 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:00.903645 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:00.907228 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.996892 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:00.998657 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.000185 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.002975 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.017756 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240501 20:00:01.025192 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.029242 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.050192 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:01.051209 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:01.089035 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.091241 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.093788 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.113090 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.113765 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.117377 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.153054 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:01.173954 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.175288 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.176312 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.180974 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.185544 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.197494 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.276088 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.279495 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.288695 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.289247 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.298025 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.301754 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.334007 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:00:01.387785 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.389135 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.390563 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.391404 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.392328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.393916 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:01.422870 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:01.433446 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:00:01.435743 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:01.449656 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.451494 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.456591 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.456822 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.460731 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.461001 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:01.510241 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 20:00:01.510886 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:01.543027 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.547245 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.547827 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.548614 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.555635 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.570645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.586496 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:01.647389 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:01.651140 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:01.758088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.758090 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.762167 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.763041 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.763670 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.772992 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:01.785966 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.853237 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.855031 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.861676 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.865275 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.865501 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.871523 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.953303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:01.954211 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:01.955492 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:01.957765 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:01.966818 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:01.968183 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:01.981997 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:02.062848 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:02.085434 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.087992 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.089349 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.095530 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.105089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.106822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:02.113670 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.115134 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:02.125455 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.133425 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.133942 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.135937 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.140997 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.155268 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.241127 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.243692 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.244405 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.244740 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.251331 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.255995 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.336835 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.338037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.343214 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.345050 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.349212 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:02.362051 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.369464 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:02.412670 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:02.415134 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:02.431877 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:02.462946 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.470103 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.470937 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.480830 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.486883 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.488677 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.489926 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.490574 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.491459 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.505499 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.507230 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.538004 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:02.562345 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.613803 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.616204 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.618896 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.625164 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.627267 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.628119 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:02.671159 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:02.694502 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.717005 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.719558 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.720579 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.725188 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.729676 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.732396 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:02.752293 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:02.765990 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.833186 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.837076 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.847934 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.853245 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.853821 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.862134 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:02.872220 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:02.881340 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:02.889133 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.890523 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.894505 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:02.897877 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:02.898699 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:02.899768 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:02.945111 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:02.990516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:02.997973 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:02.998245 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.001041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.002413 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.003090 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:03.019208 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:03.040541 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:03.049822 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:00:03.070621 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:03.104389 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.106148 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.108949 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.109848 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.112227 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.113876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.199599 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.200544 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:03.201562 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:03.203753 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.208487 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.209903 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.213059 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:03.258195 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:03.262496 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.263515 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.269084 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.270787 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.273036 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.273326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:03.306978 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:03.310277 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:03.316396 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:03.514981 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.516542 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.517534 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.521907 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.524133 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.524367 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:03.545112 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:03.609462 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.610229 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.615573 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:03.615628 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:03.627732 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.628365 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.642547 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:03.706350 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:03.713348 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.715287 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.718401 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.719903 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.721078 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.740770 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:03.791416 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:00:03.801105 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:03.835896 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.842303 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.843951 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.847389 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.854991 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.855458 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:03.867903 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:00:03.876243 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:03.877851 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:03.883445 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.885361 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.889339 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.894863 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:03.985545 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:03.988409 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:03.991617 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.001672 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.006561 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.008014 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:04.025017 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:04.078645 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:04.097745 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.103973 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.105576 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.110459 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.115947 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:04.117422 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:04.124563 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.206946 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.207278 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.210253 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.212802 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.212877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.223209 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:04.228739 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:04.256351 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:04.258474 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:04.274444 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.279170 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.280040 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.282774 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.286978 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.306989 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:04.328521 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:04.391559 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.392204 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.396509 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.397853 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.401062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.401983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.503397 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.505702 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.515594 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.516331 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.516479 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.524199 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.558575 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.565840 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.566319 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.570154 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.582080 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.592195 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:04.632861 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:04.645654 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:04.672981 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.675650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.675861 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.676813 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.688702 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.689568 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:04.693359 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:04.758383 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:04.791966 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.792580 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.795715 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.795747 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.796433 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.826390 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.896458 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.897123 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:04.898139 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:04.898483 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.908013 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.908129 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.922331 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.945966 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:04.954008 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:04.954282 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:04.962515 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:04.964866 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:04.977002 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.049582 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:05.056547 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.059567 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.060565 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:05.061486 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.065634 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.078792 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.083722 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:05.173874 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:05.215190 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:05.227082 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.227528 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.233708 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.233763 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.235870 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.247049 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:05.247198 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:05.247433 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:05.342540 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240501 20:00:05.343153 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:05.353001 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:05.380972 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.382758 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:05.384313 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:05.389086 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.391525 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.393098 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.393466 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.420850 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:05.429515 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:05.490865 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.491357 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.492841 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.494472 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.509290 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.513679 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:05.513974 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.573115 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.580528 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.581460 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.593740 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.602490 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:05.619345 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:05.622697 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.622745 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:05.653985 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.655234 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.663921 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.665446 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.665477 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:05.668778 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:05.686107 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:05.686502 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.733026 8985 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:00:05.733127 8985 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:00:05.733490 8985 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:00:05.734390 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:05.734483 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:05.734565 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:00:05.734705 8985 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:00:05.760006 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.765913 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.766817 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.767180 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.777030 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.777048 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:05.824947 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:05.863421 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.868028 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.868858 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.870090 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.871685 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.880461 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.964211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:05.972646 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:05.974785 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:05.977654 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:05.977710 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:05.982461 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:05.997550 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:00:06.015019 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.024571 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.025712 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.025730 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.027392 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:06.031495 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:06.049074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:06.049930 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:00:06.050552 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:06.079198 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:06.091707 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:00:06.136592 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.142441 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.144536 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.146347 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.165784 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.166246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:06.182543 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:00:06.241676 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:06.249760 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.249899 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.252759 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.253958 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.257584 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.261042 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.348580 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.352381 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.353663 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.357676 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.360386 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.367489 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:06.369508 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:06.425941 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.427906 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.429500 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.438925 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.439340 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:06.445493 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:06.458338 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:06.471019 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:06.522467 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:00:06.532940 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.533838 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.541198 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.542351 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.542906 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.563998 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.648114 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.654147 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.657559 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:06.658432 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:00:06.658836 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.661667 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.661880 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.708727 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.710251 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.715126 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.721284 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.721906 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:06.724465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:06.724937 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:06.738416 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:06.743324 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:06.757581 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:06.798733 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:06.803459 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:06.803982 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.805402 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:06.808478 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:06.818889 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:07.011322 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:07.021168 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.026469 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.028074 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.038523 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.038602 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.053875 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:07.085759 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.115818 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.120467 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.123373 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.127993 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.128811 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.138790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.151250 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.223765 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.226078 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.226331 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.228654 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.230229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.254307 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.280782 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:07.289930 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:07.306481 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:07.342383 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.358300 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.366084 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.366648 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.367035 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.370610 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.380143 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.398799 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.401043 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.401238 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.401293 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.412528 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.423466 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.494819 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.496963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.498668 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.501431 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.503927 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.509181 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.580826 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:07.594939 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.596917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.597023 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.598347 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.604367 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.604764 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.624418 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:07.678191 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:07.697237 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.718641 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.728204 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.729772 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.731434 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.742568 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.743144 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.755524 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.759395 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.770818 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.773761 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.778352 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:07.780642 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.788036 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:07.853853 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:07.872956 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.874842 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.874881 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:07.880316 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:07.882951 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:07.892400 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.898026 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:07.903633 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:07.935822 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:07.978912 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:07.979051 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:07.979768 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.980751 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:07.983415 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:07.984066 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:07.989754 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:08.002067 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:08.020026 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:08.041925 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:08.094303 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:08.098035 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.099310 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.113497 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.117066 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.125880 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.132994 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.137059 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.141055 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.146368 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.148296 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.151521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.155901 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:08.163043 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:08.215204 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:00:08.255729 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.256564 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.257815 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.269213 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.271310 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.271585 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:08.286047 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:08.354194 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.354892 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.365061 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.369130 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.377854 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.383212 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:08.441949 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:08.467031 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.470678 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.474056 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.478296 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.479182 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.491012 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:08.525653 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:08.526859 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.528343 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.530223 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.531284 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.533440 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.539644 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:08.622978 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:08.700547 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:08.740989 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:08.801784 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:08.806313 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:00:08.808300 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.809890 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.819746 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.822861 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.834321 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.876245 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.876667 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.882251 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.883770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.884006 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.904270 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:08.907253 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:00:08.925693 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:08.975302 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:08.975805 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:08.978340 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:08.990986 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:08.992404 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:08.993813 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.100653 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.102030 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:09.103423 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.104610 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.106320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.117677 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.131089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.172041 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.177362 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.181985 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.182981 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.183663 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.199002 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.254659 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.260321 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.260955 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.262176 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.264398 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:09.278584 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.293159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:09.317843 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:09.318457 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:09.322196 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:09.340595 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:09.346395 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.367575 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.369748 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.372960 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.375718 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.382812 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.397316 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.488480 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.492136 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.496251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.500288 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.513180 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.515947 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.553427 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.553882 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.554157 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.556710 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.563808 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.565085 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:09.593441 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.651803 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:09.652737 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.659716 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.662782 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.667161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.667165 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:09.743783 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:09.757555 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.758013 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:09.765535 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.766707 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.768002 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.769893 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:09.790939 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:09.841310 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.875499 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.881105 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:09.884193 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:09.886674 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:09.893595 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:09.896283 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.896703 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.899336 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:09.900732 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.901499 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:09.922078 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:09.926625 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.933517 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:09.933773 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:09.944572 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:09.977766 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:10.023149 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.024231 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.026391 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.034147 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.039326 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.042469 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:10.073784 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:10.128541 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.133216 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.133615 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.133944 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.143018 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.162246 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:10.171715 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:10.247139 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.255228 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.260557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.261059 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.263979 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.264865 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.267014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.268829 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.282858 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.290025 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.291754 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:10.299500 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:10.315871 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:10.387873 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 20:00:10.406176 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:10.456113 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:10.467940 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:10.474022 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:10.515116 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:10.552340 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:10.579252 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:00:10.590744 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:10.604682 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.618666 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.620103 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.621289 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.621618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.634429 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.654003 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.660073 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.660671 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.669817 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.673194 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.674376 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:10.700667 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:10.739480 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:10.757875 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.758612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.764568 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.766324 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.773986 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.786533 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:10.792182 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:10.866281 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.868005 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:10.870684 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.879083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.882491 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.882768 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:10.946975 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:10.969632 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:10.971740 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:10.973361 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:10.981062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:10.984362 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.002015 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.029019 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.032267 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.035835 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.038817 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:11.046352 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:00:11.053547 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:11.055272 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.061753 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:11.093590 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:11.109316 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:00:11.124047 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:11.140372 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.141009 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.143416 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.148582 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.151363 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:11.152199 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:11.154790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:11.194345 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:11.249856 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.255976 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.257457 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.262985 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.265154 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.275113 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.340839 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.342051 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.345208 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.352389 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.357419 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:11.366585 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:00:11.372721 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:11.406380 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:11.409525 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.414973 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.416762 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.418074 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.418239 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.435500 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.510659 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.511312 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.522539 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.523324 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.524705 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.525512 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.623710 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.628182 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:11.631466 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:11.633220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.638928 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.639104 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.655807 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:11.674381 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:00:11.700229 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.700701 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.706161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.706954 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.710407 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:11.713637 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:11.725950 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.780457 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:11.781558 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.784504 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.789150 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.794174 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.801949 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:11.817493 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:11.855350 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:00:11.876686 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.881690 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.886267 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.887275 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.889678 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.901571 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:11.925493 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:11.963189 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:11.984298 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:11.984601 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:11.990082 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:11.991536 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:11.993666 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.000816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:12.026784 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:12.057670 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.059501 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.065374 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.068861 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.071650 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.093734 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:12.164472 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:12.172279 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:12.194305 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:12.261708 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:12.290419 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.291977 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:12.292732 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:12.297879 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.309497 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.313618 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.321767 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.391479 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.392412 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.395663 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.404305 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:12.409075 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:12.410861 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.419426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.457753 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.459582 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:12.459890 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:12.459899 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.463204 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.470851 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:12.479285 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:12.481343 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.562827 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.563678 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.564110 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.565335 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.574157 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.578785 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:12.596668 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:12.621421 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:12.660403 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:12.677767 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.681915 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.686542 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.690039 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.695300 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.695744 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:12.736557 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:12.751767 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.758163 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.758491 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.767307 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.773556 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.787866 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:12.797752 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:12.840600 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.842542 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.843838 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.845381 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.848071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:12.856019 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:12.858570 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:12.871266 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:12.894054 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:12.949954 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:12.952518 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:12.960220 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:12.960256 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:12.960798 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:12.963294 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:12.973555 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:13.065202 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.073305 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.073457 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.075533 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.077077 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:13.089318 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:13.094121 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.130973 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.131733 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.134049 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.134397 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.156998 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.158562 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:13.187322 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:13.206243 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:13.227173 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:13.230932 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.231400 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.233668 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.236263 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.239398 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.248018 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:13.249176 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:13.250299 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:13.337335 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.339258 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.341207 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.344063 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.345078 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.357344 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.458317 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.465072 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.466989 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.468669 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.477329 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.480827 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.484122 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.495820 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.495882 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.503461 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.518673 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.539575 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:13.553630 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:13.592589 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.592828 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.595909 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.601770 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.609949 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.612612 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:13.630095 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:13.678133 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:00:13.679639 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:13.685274 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:13.699885 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.701027 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:13.701920 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:13.706368 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.711251 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.718078 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:13.783560 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:13.820850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:13.825755 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:13.829902 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.831029 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:13.833029 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:13.847714 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:00:13.848057 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:13.972879 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:14.021607 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.022625 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.031544 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.034164 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:14.038934 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.044270 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.045797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:14.118115 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:14.146847 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.166494 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.166621 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.172618 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.178182 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.187836 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.198809 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.205384 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.205646 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.210709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.216598 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.230880 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.238116 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:14.272030 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.300318 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.301606 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.306283 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.308122 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.313763 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.314244 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:14.377664 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.408663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.408717 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.413748 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.417809 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.419323 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.427161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:14.466234 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:14.467093 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.512030 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.513545 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.515590 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:14.515661 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:14.521036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.526170 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.529003 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.572732 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.577294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.578831 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.588963 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.598105 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.604290 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.677453 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.680797 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.682233 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.682512 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.683387 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.697734 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:14.698144 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:14.758643 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.781116 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.782081 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.783820 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:14.787114 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.788861 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.808266 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:14.840045 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:14.877563 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:14.907923 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.911828 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:14.914774 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:14.919868 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.922714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.925334 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:14.928377 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:14.929466 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.946487 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:14.948348 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:14.969717 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:14.972002 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:14.973996 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:14.978262 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:14.987361 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:15.041982 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:15.054736 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:15.055119 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:15.072470 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.076875 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.077000 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.077280 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.081096 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.095703 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:15.099972 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:15.132391 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:15.181766 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.183382 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.185310 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.187764 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.188427 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.198079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:15.261287 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:15.284863 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.286357 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.287586 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.292300 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.311286 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.311548 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.316406 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:15.320169 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:15.328871 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.329890 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.343798 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.358137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.359335 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.442507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:15.446405 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:00:15.447436 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.449462 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.449584 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.450449 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.450798 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:15.484956 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:15.485877 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:15.535115 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:15.610755 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:15.650185 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.650369 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.651058 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.654350 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.657706 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:15.659606 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:15.666082 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.743927 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.747372 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.748344 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.749222 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.751857 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.753057 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.842626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.843006 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.844071 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.844880 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.853035 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.869946 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:15.893561 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:15.913985 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:15.948729 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:15.952821 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:15.959429 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:15.960275 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:15.964130 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:15.964305 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:15.972215 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.011137 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.012141 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.013445 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:16.016669 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:16.020226 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.025868 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.033756 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:16.037945 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:16.104318 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:00:16.104918 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:16.119086 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.119216 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.120153 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.121073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.130033 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.130409 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:16.131052 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:16.138787 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:00:16.162807 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:00:16.219883 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.222216 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.222462 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.226228 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.233745 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.236160 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:16.266527 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:00:16.340291 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.342304 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.343858 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.346743 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:16.357465 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:16.365170 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.367650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.374490 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.378818 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.379171 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.383328 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.390276 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.392689 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.469635 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:16.488966 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.491019 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.492242 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.492646 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.504565 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.505421 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.556969 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:16.587014 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.590301 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.590876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.591421 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.594534 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.603360 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.649979 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:16.674410 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 20:00:16.703773 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.716607 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.717027 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.717301 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.723919 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.724294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.725805 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.738830 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.742127 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.746281 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.756742 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.765754 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.847733 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:16.859601 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:16.862445 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:16.864344 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.865537 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.865902 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.867879 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.872406 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:16.971789 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:16.975198 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:16.975965 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:16.981801 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:16.982473 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:16.986316 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:16.992535 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:17.009512 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:17.017684 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:17.032563 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:00:17.078066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.078557 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.083297 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.084014 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.089630 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.092622 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:17.118619 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:17.138695 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.148017 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.148562 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.154575 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.162010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.164995 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.175055 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:17.387435 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.387929 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.388176 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:17.390913 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.391695 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.392853 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:17.393153 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:17.397713 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.401975 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:17.402539 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:17.442193 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:17.474834 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:17.479566 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.483146 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.485456 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.485628 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.486148 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.486877 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:17.503710 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.572120 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.575920 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.577416 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.579535 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.583052 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.592124 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:17.600944 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:17.645776 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:17.687840 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:17.694234 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:17.696475 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.696939 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.713477 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:17.716848 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:17.717144 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.717475 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.724390 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.732426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.735324 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.735709 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.736359 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.761706 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.764068 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:17.837026 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:17.851328 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.852223 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.854048 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.857570 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.860371 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.860797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:17.935952 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:17.952512 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:17.953128 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:17.954201 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:17.955379 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:17.958181 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:17.959427 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.068516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.082624 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.084025 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.084897 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.085723 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.088644 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:18.091693 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:18.116065 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.117467 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.122125 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.122509 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.122502 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:18.133592 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:18.135164 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.231238 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.231490 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.232271 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.235337 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.245406 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.250646 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:18.253474 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:18.296818 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:18.319087 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:18.331080 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.332131 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:18.333304 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:18.340438 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.343362 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.344591 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.353096 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:18.407311 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:18.450139 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.453591 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.462939 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.465883 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.469653 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.486719 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.489044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.494868 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.499572 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:18.502400 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:00:18.507525 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.508543 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.511086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:18.543359 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:18.581887 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 20:00:18.620973 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.621831 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.622963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.627310 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.634625 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.641481 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:18.692471 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:18.692683 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:00:18.704300 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:18.717262 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.718703 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.721750 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.722476 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.724756 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.728509 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.836248 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.838408 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.839807 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:18.844851 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.847793 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:18.849313 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:18.855671 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.859094 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:18.879978 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:18.893640 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:18.895774 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.899078 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:18.909106 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:19.077454 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:19.095008 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.206463 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.206485 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.206939 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.213387 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.221823 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.231216 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.237819 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:19.238049 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:19.247818 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.262533 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.264106 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.265062 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.265208 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.351913 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.353463 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.353859 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.363338 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.365841 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:19.380460 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.381034 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:19.406898 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:19.414129 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.459141 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.460805 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.463012 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.465701 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.476032 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.487103 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.522881 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:19.535070 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.563124 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.572711 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.572870 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.574038 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:19.589417 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.592887 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.600335 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.600620 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.607183 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.621017 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.621551 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.630736 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.645085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.721388 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:19.722335 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.724545 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.727414 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.730291 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.735193 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.740728 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.827078 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.829846 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.834657 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.835755 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:19.837276 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.838752 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.839679 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.844808 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:19.870847 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:19.881440 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:19.939803 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.941545 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.941967 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:19.942431 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:19.943714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:19.954044 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:19.968727 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:19.977716 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:19.982241 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:19.993053 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:19.997076 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.003379 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.003988 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.007476 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.068791 13987 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:00:20.068887 13987 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:00:20.069294 13987 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:00:20.070259 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:20.070425 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:20.070500 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:00:20.070642 13987 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:00:20.096292 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:20.098636 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:20.099905 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.106988 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.108439 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.115924 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.116859 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:20.201149 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.202489 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.204113 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.207551 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:20.213707 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.216100 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:20.220311 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.226779 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:20.238639 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:20.296037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.299504 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.300798 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.304508 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:20.314433 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:20.315842 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.319914 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.367226 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.372383 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.375887 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:20.378182 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.397480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.398275 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.461202 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 20:00:20.470549 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.471498 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.473134 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:20.473924 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.482074 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.492010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.511631 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:20.541255 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:20.567276 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:20.572957 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.576987 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.577065 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.577333 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.581183 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:20.586762 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:20.597405 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:20.656628 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:20.656718 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.657578 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.662134 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:20.668397 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.669292 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.676158 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.782697 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:20.834626 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:20.893033 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:20.895584 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:20.900533 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:20.901324 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:20.908820 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:20.913975 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:20.926656 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:20.932562 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:20.996043 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.000365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.002606 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.004083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.010977 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.025063 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:21.039058 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:21.063254 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.064977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.065953 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.067575 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.070724 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.097400 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.175980 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.176074 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.176915 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.192487 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.195116 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.199503 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:21.214329 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:21.222357 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:00:21.287106 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:21.287468 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.287844 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.289605 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.292320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.306355 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.309396 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:21.312971 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:00:21.357374 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:00:21.385447 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.390317 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.394116 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.396453 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.396505 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:21.418699 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:21.426805 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:21.444062 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:00:21.452085 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:21.455888 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.456612 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.461529 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.464272 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.466524 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.481261 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:21.554458 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:21.562136 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.562252 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.566056 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.567360 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.573499 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.575671 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.668395 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.671146 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.674355 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.674672 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.679771 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.681612 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:21.733399 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:00:21.739586 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:21.762810 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.765457 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.769274 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.770409 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.778791 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.780333 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.823457 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.831413 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.833185 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.834059 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.836139 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.844720 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.938891 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:21.938894 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:21.939025 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:21.941136 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:21.949482 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:21.953199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:21.962342 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:21.982993 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:22.000200 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:22.044828 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.045187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.047703 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.050951 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.057510 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.058750 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.072929 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:00:22.084187 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:22.142635 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.146329 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.148171 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.152335 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.168386 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.175242 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.183494 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.189429 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.197650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.205168 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.217096 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.221081 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.222355 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.228516 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.288632 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.310285 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.316766 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.317232 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:22.317448 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.318637 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.326167 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.338831 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.408340 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:22.408969 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:22.430070 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.484068 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.484099 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.488509 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.489130 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.499768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.505836 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.518494 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:22.519336 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:22.530486 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:22.569963 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:22.610062 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.620388 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.620827 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.623222 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.624847 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.634933 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.635622 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.718873 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.722098 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.722519 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.722965 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.723212 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.746398 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:22.778705 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:22.782001 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.818455 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.819809 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.824332 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:22.840090 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.846402 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.850126 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.887919 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:22.888530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.889904 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.889972 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:22.897521 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.902248 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:22.932204 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:22.948082 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:22.987939 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:22.993028 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:22.994582 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:22.998910 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:22.999210 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.000319 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.005851 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:23.080296 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:23.094043 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.096304 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.096307 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.099244 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.099756 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.100164 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.124266 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:23.191408 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:23.199174 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:23.201138 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.201705 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.204532 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.207310 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.212648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.217744 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.256747 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.261010 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.262446 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.266363 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.266922 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.270448 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.368450 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.370931 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.371351 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.373155 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.373544 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.378217 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.474674 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.477914 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.480809 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.489527 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.490833 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:23.503252 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:23.506444 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.518451 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:23.558075 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.558140 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.561527 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.561515 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.578438 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.578939 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.641856 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.642320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.645892 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.647954 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.648161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.649909 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:00:23.665690 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.666841 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:23.736872 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:23.742110 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.743363 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.746892 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.753139 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.757975 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.773739 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:23.776891 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.852161 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.858531 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.860651 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:23.861235 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.864004 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.864228 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:23.914510 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:00:23.923790 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.926069 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:23.926597 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:23.934494 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:23.941737 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:23.954437 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.008610 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.008946 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.009663 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.017174 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.018493 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.019322 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:24.033432 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:00:24.057659 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:24.071020 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:24.104507 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:24.240491 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.240541 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.240799 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.243528 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.244468 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.249050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:24.252683 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.332402 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.336884 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.339397 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.339810 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.341123 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.369417 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:24.410565 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.441560 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.443284 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.447237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.447593 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.456811 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.469409 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.557574 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.561995 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.562426 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.574062 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:24.580137 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.580749 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.580865 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.582988 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.588457 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.588748 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.590353 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:24.613809 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:24.615190 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.616164 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:24.648260 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:24.711896 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:24.712637 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:24.714988 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.719676 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.720140 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.721207 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.721320 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.723922 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.808107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.813045 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.820086 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.821974 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.823752 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.824014 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:24.850384 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:24.915503 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:24.927034 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.943732 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:24.956365 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:24.956971 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.959050 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:24.960263 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.961441 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.978389 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:24.986739 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:24.990372 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:24.992986 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:24.994798 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:24.995281 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:25.007859 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.019285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:25.027423 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:25.056627 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:25.086896 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:25.101217 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.101627 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.104144 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.106591 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.114094 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.115885 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:25.136984 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:25.154772 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:25.186156 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:25.209137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.212180 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.213070 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.213752 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.222117 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.223935 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:25.229591 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:25.302189 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:25.329113 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.329524 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.332742 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.338474 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.339864 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.341300 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.352697 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.364372 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.364475 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.380296 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.380554 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.388724 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.483975 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.484575 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.489418 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.493331 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.497072 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.506724 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:25.527949 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 84 similar messages]
I20240501 20:00:25.577632 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.580860 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.582558 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.585713 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.593199 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.593276 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:25.609606 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:25.687078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.691298 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.694293 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.697024 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.699232 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:25.714406 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:25.724118 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:25.738644 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:25.739425 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:25.742229 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:25.747107 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:25.751909 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:25.757953 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:25.760497 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:25.780409 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:00:25.812429 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:26.026746 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.029616 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.032734 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.046691 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.049587 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.064194 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:26.082460 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:26.103655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.105669 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.107650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.108399 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.122450 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.123991 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:26.133718 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:26.202335 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.206548 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.207948 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.209952 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.213421 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:26.214672 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:26.215029 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:26.274667 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:00:26.275278 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:26.301329 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.305040 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.309564 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.310446 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.322121 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.329085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:26.334565 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:26.353186 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 20:00:26.354018 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:26.407495 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.414294 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:26.415839 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:26.416378 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.420771 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.441656 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.442489 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.471391 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.475677 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.478796 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.478981 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.479441 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:26.488271 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:00:26.507149 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:26.517113 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:26.578002 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.578114 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.580272 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.581311 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.581887 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.608381 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.685556 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.687788 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.693809 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.696216 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.696449 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.696671 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:26.769552 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:26.775414 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.778218 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.778448 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.782152 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.782825 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.793251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:26.810273 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:26.841759 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:00:26.844190 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.850375 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.852169 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.853011 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.854285 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.858067 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:26.873308 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:26.953378 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:26.953429 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:26.956796 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:26.957084 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:26.957463 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:26.959887 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.056138 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.060124 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:27.060607 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:27.068457 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.076928 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.079650 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.084600 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.136783 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.141502 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.141916 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.144282 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:27.150209 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:27.164566 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.178414 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:27.196993 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:27.205536 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:00:27.221038 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.223311 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.225875 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.228108 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.229676 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:27.231262 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:27.231870 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:27.237565 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:27.288208 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:27.325330 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.326529 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.331283 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.333081 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.340826 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.346101 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:27.362915 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:27.380151 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:27.395718 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:27.414952 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:27.431278 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:27.443236 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.443954 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.446012 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.449265 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.454203 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.478152 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:27.494076 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:27.504755 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.506302 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.510079 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.517457 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.523087 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.537024 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:27.591315 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:27.626410 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:27.633628 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:27.712726 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:27.759804 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.760730 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.764971 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.767208 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:27.775856 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:27.779245 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.786217 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:27.797505 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:27.879849 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.880419 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.885582 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.886821 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.889511 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:27.893890 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:27.901589 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.905155 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:27.915534 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.918535 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:27.930871 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:27.948047 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.026751 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.029883 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.030923 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.031397 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.035204 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.044778 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:28.065078 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:28.140352 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.144768 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.145037 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.145627 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.147044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.154901 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:28.190101 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:28.201781 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:28.241904 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:28.243377 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.249055 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:28.251863 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:28.252184 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.254144 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.262171 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.276513 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.306687 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.308781 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.310067 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.311148 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.312361 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.312613 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:28.390345 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:28.425550 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.426296 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.430768 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.431877 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.450093 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.450199 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:28.500260 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:28.530632 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.531806 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.532060 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:28.542228 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:28.543530 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.556351 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.557554 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.621417 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.626590 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.629166 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.630472 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:28.632404 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:28.638682 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.655038 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.692693 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.695282 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.695859 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.698503 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.700726 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.710376 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.799089 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:28.799492 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:28.799899 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.799994 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.802601 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.803968 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.816557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:28.821130 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:28.821440 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:28.871320 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:00:28.897204 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:28.916692 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.918416 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.924644 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:28.933585 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.934554 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:28.937156 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:28.952883 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:00:28.986423 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:28.994612 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:28.997833 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:28.998900 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:28.999401 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.000013 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.009824 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.083014 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.083258 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:29.083559 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.095067 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.102883 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.105645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:29.193588 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:29.198746 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:29.200364 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.202159 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.204228 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.205287 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.213183 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:29.286211 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:29.302368 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.303349 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.303943 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.310535 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:29.312518 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.315361 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:29.433493 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.500034 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.500365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.505807 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:29.507761 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.510928 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.511168 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:29.519949 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:29.537962 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:29.558007 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.613413 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.617957 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.630470 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.641048 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.641671 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.643294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.644177 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:29.672619 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.674162 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.675529 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.686249 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.687919 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:29.723544 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:29.745968 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:29.755201 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:29.764024 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.769076 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:29.771246 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.771759 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.777143 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.778447 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.787179 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:29.831419 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:29.838353 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.878765 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:29.882608 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:29.882705 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:29.883888 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.886533 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:29.893833 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:29.908082 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:29.918545 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:29.957302 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:29.998586 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.000568 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.001263 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.004329 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.019050 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.021661 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.027323 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:30.040481 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:30.041112 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.044569 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.047442 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.067759 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.071949 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.100921 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:30.153956 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.157713 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.157872 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.159413 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.171639 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.178786 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.198108 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:30.239549 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:30.262005 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.264634 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.265944 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.266379 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.266521 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.274116 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:30.307931 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:30.361500 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:30.365757 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:30.366243 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.368872 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.374369 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.377017 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.401367 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.432066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.433701 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.439250 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.448271 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.448706 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.458495 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.533178 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.535319 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.537034 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.539666 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.540078 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 20:00:30.540680 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:30.545543 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.559716 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.645044 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.649451 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.650195 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.654045 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.654851 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.666692 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.710772 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:30.727350 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.729389 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.732535 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.735292 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.735715 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.738288 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:30.748303 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:30.781975 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:30.784503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.785132 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.785998 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.800369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.800474 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:30.806056 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:30.809403 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.890473 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:30.894523 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:30.894903 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:30.896466 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:30.899296 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:30.904670 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:30.919001 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:30.931182 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:30.955385 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:31.055313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.061851 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.062929 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.066036 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.066345 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.066856 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:31.133430 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:00:31.164573 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:31.200570 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.203791 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.206997 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.207260 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.215245 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.221185 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:31.302879 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:00:31.303803 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:31.304697 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.306607 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.306854 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.311328 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.317826 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.329700 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:31.363279 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:00:31.401158 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.405947 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.406244 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.414054 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:31.414829 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:31.415494 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:31.416781 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.417863 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.461246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.469347 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.472740 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.473965 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.485807 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.489012 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.566792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.569370 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.570284 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.573159 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.576689 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:31.582553 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:31.597355 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:31.608224 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:00:31.672598 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.677670 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.678599 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.679962 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.680567 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:31.695385 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:31.697435 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:31.708858 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:31.764705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.772210 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.776019 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.776098 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.779124 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.798050 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:31.823602 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:31.830045 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.830061 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.830477 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.831708 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.851117 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.852663 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:31.878576 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:00:31.879323 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:31.929374 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:31.931409 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:31.933318 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:31.941071 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:31.944754 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:31.947788 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:32.021620 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:32.032650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.033247 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.037328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.046242 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.047392 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.052934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.141856 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.144680 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.145027 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.148850 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.149477 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:32.151324 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:32.156157 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:32.172981 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:32.196220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:32.203002 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:32.206665 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.206889 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.207800 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.214774 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.221061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:32.241783 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:32.244360 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:32.277329 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:32.303967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.307951 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.314462 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.314631 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.314904 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.324781 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:32.327457 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:32.340384 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:32.402598 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:32.410420 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.416025 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.416066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.417441 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.418829 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.419088 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:32.448948 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:32.496156 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:32.496542 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:32.501225 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.502317 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.502836 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.504654 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.508226 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.508756 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:32.522325 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:32.526211 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:32.552639 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:32.565837 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.568243 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.569643 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.571400 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.578456 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.582806 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:32.668090 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:32.804893 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.808776 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.812031 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:32.813910 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.823781 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.831000 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:32.872115 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:32.902402 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:32.908152 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.908190 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:32.913816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:32.914188 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:32.915688 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.000715 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.001120 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.003207 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.003836 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.007052 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.016323 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:33.040623 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:33.117952 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.129163 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.130102 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.131067 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.136238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.142010 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:33.146569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.166054 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:33.169533 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:33.170549 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.170828 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.179842 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.202797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.203498 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.207669 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:33.210155 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:33.274008 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.280416 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.288813 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.291239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.297518 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.297675 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.381013 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.382328 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.384667 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.387612 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.388803 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.401386 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:33.418035 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:33.491559 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:33.505398 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.505641 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.516011 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.516211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.516856 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.526105 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.534444 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:33.535866 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.535979 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.543962 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.546348 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.571859 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.572098 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:33.582126 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:00:33.643170 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:33.651865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.653666 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.656311 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.658728 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.661069 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.674371 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.745186 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:33.758147 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.760169 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.761668 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.763700 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.769444 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.781934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.825817 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:00:33.879078 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.884438 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.886395 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.887130 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.889140 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:33.904378 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:33.915933 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:33.916458 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:33.916602 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:33.919607 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.924237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:33.936128 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:33.936385 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:33.941998 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:33.955775 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:34.031190 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.033950 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.034950 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.036156 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.045027 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.045886 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.046134 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:34.072253 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:00:34.134654 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.140233 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.140686 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.141750 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.148275 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.150995 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.246265 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.249269 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.249421 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.252123 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.263635 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.288560 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.295476 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.298043 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.299379 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.304421 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.311734 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.317004 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.358573 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:34.424500 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:34.471850 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:34.534508 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:34.584162 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.628252 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.628877 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:34.629045 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.635293 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.636713 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.641722 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.647786 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.651314 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.652838 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.661120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.670599 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:34.680734 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.681584 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.682833 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.703877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.717417 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.775435 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:34.777462 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.779541 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:34.782877 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.789064 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.798440 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.856608 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:34.886906 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:34.889091 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.891557 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.892935 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:34.895515 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:34.897465 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:34.900194 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:34.901546 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:34.903172 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:34.914500 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:34.925061 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:34.933382 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:34.972522 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:35.004339 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.006073 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.007073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.008546 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.013023 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.028187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.079229 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.079753 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.085104 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.089169 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.092749 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.093087 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.140326 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:35.184018 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.185446 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.187033 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.190897 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.191120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.204597 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.295331 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.296581 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.301877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.302294 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.302981 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.309387 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:35.315011 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.363929 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.371634 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.374337 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.375341 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.382153 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.382158 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:35.447952 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:35.453554 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.454540 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.458271 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.459230 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.461377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.468255 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.475328 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:35.542356 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 20:00:35.551898 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.554751 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.555208 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.558944 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.568457 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.570467 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.593772 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:35.608258 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:35.612843 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:35.671422 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.672047 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.678480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.681327 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.684065 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.685874 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.738209 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.741859 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.742817 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.749053 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.749356 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.756037 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.822872 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.823482 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.826139 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.826336 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:35.827446 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.832326 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.836717 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:35.917076 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:35.917841 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:35.923981 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:35.925346 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:35.925662 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:35.933053 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:35.937729 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:35.965323 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:36.030328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.031734 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.035650 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.040164 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.042891 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.066113 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.107141 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.108345 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.111342 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.111757 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.112422 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.125895 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:36.244762 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:36.253829 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:36.273633 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:36.307562 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:00:36.309021 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:00:36.318523 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:36.350303 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.354454 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.356253 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.359462 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.362707 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.381222 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:36.423301 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:00:36.424259 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:36.440110 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.441823 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.443142 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.451025 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.451386 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.457118 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:36.488831 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:36.511898 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.514217 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.515621 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.516335 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.517211 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.535594 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:36.537876 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:36.616142 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.618193 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.618507 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.632330 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.632885 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.633961 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:36.659232 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:00:36.736997 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.737736 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.741561 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.748697 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.749105 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.756326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.795351 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.800014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.802034 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.805127 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.819737 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.831614 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.881114 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.881999 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.884948 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:36.887157 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.897122 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:36.897480 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 20:00:36.905082 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:36.905622 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:36.935052 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:36.955528 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:36.983019 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:36.983337 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:36.986860 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:36.987036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:36.994187 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.001766 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:37.032110 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:37.035426 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:37.043217 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:37.105257 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.106159 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.109195 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.110239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.117175 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.124433 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.170590 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.171017 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.176785 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.177165 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.181788 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.184720 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:37.202131 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:37.222178 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.265537 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:37.266566 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.267156 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.267882 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.269088 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.270831 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.271944 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:37.303875 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.371276 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.375919 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.378389 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.379566 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.383608 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:37.385812 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:37.397533 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:00:37.400487 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.403342 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:00:37.466500 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.501770 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.503269 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.512209 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.514565 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.515398 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.516436 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.517419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.518155 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.519047 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.533524 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:37.558825 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.558930 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.562752 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.578706 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:37.580612 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:37.642544 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.643792 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.644534 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.646755 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.650194 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.656360 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.678936 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.744133 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.744688 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.748859 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.757025 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.766001 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:37.766410 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.769273 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:37.825325 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:37.846589 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:37.847072 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:37.850793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:37.851154 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:37.862519 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:37.879971 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:37.900368 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:38.030634 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:38.055135 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.058905 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.061959 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.062234 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.069823 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.079089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:38.169358 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.180073 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.183205 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.190649 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.192292 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.192996 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.193915 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.202301 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.203951 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.206895 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.243399 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.243897 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.248020 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:38.293542 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:38.329694 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.331938 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.335325 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.337074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:38.339190 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.339264 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.339321 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:38.361994 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:38.403301 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.446348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.451040 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.454510 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.454504 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.454682 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.456812 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:38.466301 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:38.466997 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.499305 23891 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:00:38.499439 23891 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:00:38.499910 23891 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:00:38.501165 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:38.501245 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:38.501312 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:00:38.501427 23891 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:00:38.544745 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:38.568935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.569463 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.570691 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.570981 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.571949 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.576994 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.577353 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.589128 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.590728 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.614605 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:38.621711 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.624779 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.630965 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.717871 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.717871 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.723093 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.727860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.730482 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.730911 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:38.737897 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:38.819494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.825120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.826252 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.826391 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:38.826970 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:38.835799 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.845546 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:38.912783 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:38.929536 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.941803 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.947086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.947764 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:38.949352 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.959717 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.965507 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:38.968186 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:38.970489 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:38.983757 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:38.986976 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:38.995380 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:39.001217 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.088084 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.088819 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:39.098187 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:39.100739 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.104482 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.115891 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.116601 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:39.188771 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:00:39.197229 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.199391 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.201510 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.202889 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.203346 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.215113 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.309839 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.312872 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.313740 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.315830 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:39.318208 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:39.327746 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.328866 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.328959 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.329438 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:39.330785 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:39.333386 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.334403 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.364652 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.367591 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:39.394546 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:00:39.464728 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.465447 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.465481 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.468786 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.471843 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:39.476563 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:39.481359 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:39.573071 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:39.593201 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:39.622917 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:39.623471 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:39.656520 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.659255 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.660298 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.667366 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.671586 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.682237 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:39.702572 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:39.709160 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:39.732230 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:39.754916 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:39.759796 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:39.770123 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.771793 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.771840 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.774861 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.781059 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.789207 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.867635 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:39.867713 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:39.869247 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.871677 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.874730 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.889467 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.968531 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:39.971289 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:39.976119 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.978032 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:39.981124 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.000779 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.008821 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.011612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.016880 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.023416 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.039515 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:40.049870 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:40.061784 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:40.087834 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:40.101745 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:40.119953 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.137259 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.139075 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:40.139676 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.141753 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.145071 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.148159 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.158924 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:40.174136 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:40.242336 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.247908 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.247915 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.253046 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.253309 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.255173 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:40.332619 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:40.340628 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.344885 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.350895 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.353065 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:40.357825 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.359622 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.377584 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.404830 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.411518 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.417472 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.417804 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.426111 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.426923 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.506417 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.510659 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.512178 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:40.512509 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:40.513046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:40.527354 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.531344 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.536569 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:40.565493 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 20:00:40.617516 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.617995 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.618947 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.629637 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.633026 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.638629 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.712692 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.714601 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.714964 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.721422 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.732261 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.732276 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.783094 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.783799 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.784693 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.785851 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.786062 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.786734 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:40.824168 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:40.825573 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:40.833712 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:40.839612 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.883910 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.885223 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:40.888396 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.889884 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.895727 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.898178 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:40.906121 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:40.990840 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:40.995005 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:40.997262 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:40.997803 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:40.999393 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.006106 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:41.035802 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:41.075412 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.078442 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.083703 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.085842 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.086604 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.097324 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:41.130225 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:41.155900 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.156739 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.157298 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.165077 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.168846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.169242 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:41.287263 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:41.309569 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:00:41.317281 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:41.327919 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:41.383333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.385632 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.385638 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.388314 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.391863 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.391880 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:41.456050 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:00:41.476044 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.479156 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.479825 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.486775 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.487593 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.495383 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.569411 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.573472 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.574842 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.577531 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:41.587551 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:00:41.588455 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:41.593415 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.599083 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.694631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:41.694728 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:41.704567 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.704927 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.706815 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.713354 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.718034 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.720965 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.736538 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.736747 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.751484 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.753715 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.767155 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:41.821578 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:41.842846 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.849838 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.850042 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.859130 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.861694 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.863752 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:41.929586 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:41.943187 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:41.944988 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:41.945298 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:41.946761 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:41.953243 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:41.955687 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:41.965502 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:42.012777 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:00:42.014892 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 20:00:42.045868 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.061012 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:42.061272 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:42.071839 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.080322 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.080570 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.081393 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.087246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.093824 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.105513 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.109402 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.113045 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.115188 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.136025 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:42.178674 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.207446 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.212783 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.215204 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.215978 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.216799 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:42.223048 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:42.224787 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.228556 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:42.262136 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:42.269738 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.312950 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.313313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.317133 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.317992 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.324476 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.325156 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:42.358589 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:42.362267 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:42.380520 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:42.417485 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:00:42.432088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.432333 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.438467 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.447046 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.462086 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:42.462458 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:42.469391 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.472352 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.480008 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.483697 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.485862 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.487941 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.491436 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:42.517753 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:42.566162 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.578768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.585515 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.586165 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.598295 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.598500 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.615561 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:42.658078 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:42.679308 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:42.686424 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.686439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.689882 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.699713 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.700162 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.701913 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.791643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.794019 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.811717 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.813031 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:42.813621 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.814182 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.815842 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:42.823935 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:42.834256 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:42.858076 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:42.859269 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:42.866021 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:42.873772 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:42.907120 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:43.092456 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:43.136258 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:43.147548 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:43.148865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.153458 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.158717 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.165201 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.169324 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.170878 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.202767 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.204310 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.207377 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.208101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.209791 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.214946 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:43.222137 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:43.285758 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:43.300294 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.302991 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.304829 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.310987 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.324020 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.324609 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.414155 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.418294 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.418294 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.418679 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.431003 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.433960 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:43.507115 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:43.526377 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:43.532022 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:43.533886 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.535442 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.538996 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.545243 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.549762 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.557349 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.567167 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:43.572206 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:43.572496 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.573982 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.585816 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.598197 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:43.604060 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:43.676347 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.677958 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.678454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.686584 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.686909 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.698312 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.774938 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.775836 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.776291 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.776999 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.779747 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.780737 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.895011 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.898607 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:43.899857 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.905000 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.906649 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:43.907469 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:43.914098 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:43.914202 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:43.914561 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:43.919481 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.919696 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:43.925652 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:00:43.931195 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:43.954000 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:43.964458 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:43.970703 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:44.032164 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.032811 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.037799 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.041247 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.046831 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.047799 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:44.141029 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.141299 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.142290 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.146811 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:44.152998 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:44.158236 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.159049 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.261468 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.265808 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.269048 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.269228 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:44.270710 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.273938 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.274811 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.277983 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.283986 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.298924 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:44.301888 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:44.308676 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:44.310861 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.317520 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:44.373736 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:44.396461 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.397449 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.406049 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.406749 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.408851 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:44.413522 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:44.424924 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:44.455008 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:00:44.476195 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:44.502858 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.506872 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.507108 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.509241 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.515938 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.520160 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:44.566004 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.570855 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:44.598029 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.604607 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:44.609773 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.610584 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.613780 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.617045 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:44.622102 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.648142 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.676060 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:44.703951 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.714469 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.749939 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:44.755460 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:44.808596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.810703 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.813555 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.821233 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.823328 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.833189 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:44.852718 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:44.881567 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:44.927600 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.932525 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:44.933382 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.934065 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:44.940011 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.957506 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:44.961356 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.961861 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:44.981365 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:44.984792 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:44.985406 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.010046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.102365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.102823 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.108353 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.109737 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.115077 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:45.120756 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:45.133082 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:45.146697 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:45.205523 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.208299 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.215425 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:45.217438 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:45.226022 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:45.229269 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.239641 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:45.294651 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:45.313004 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.313956 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.317662 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.322674 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:45.325969 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.333663 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:45.361963 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:45.376681 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.376937 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.379657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.385205 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.386147 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.388557 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:45.416160 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:45.445454 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:45.472433 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.472683 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.477739 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.483316 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.486549 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:45.497512 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.572582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.574890 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.575135 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.576182 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:45.582965 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.585263 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:45.595602 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240501 20:00:45.613451 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:45.664855 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.665463 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.668043 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.673035 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.682399 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.683758 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:45.705296 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:45.738293 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.741070 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.743073 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.746881 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.747221 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.751475 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:45.808873 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:45.840536 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.844081 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.846541 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:45.846585 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.855650 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.855980 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.943897 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:45.944989 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:45.952545 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:45.953620 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.953864 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:45.962074 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:45.997601 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:46.015913 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:46.024448 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.026407 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.035041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.035182 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.035226 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.049978 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:46.062366 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:46.085763 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:46.103643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.105463 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.107354 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.114022 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.116133 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:46.127077 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:46.133934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.205165 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.211244 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.211869 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.214063 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.214059 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.218559 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:46.279470 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:46.314914 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:00:46.338184 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.342221 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.344239 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.344271 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.349047 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.358536 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:46.386967 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:00:46.438753 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:46.507629 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.512631 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.514045 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.514381 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.516373 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.521559 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:46.541496 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:46.608230 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.611502 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.623883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.626968 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:46.629850 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:00:46.630851 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.643325 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.685590 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.689344 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.689504 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.691390 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.703383 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.709626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:46.715461 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:46.782665 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.783608 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.783706 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.783797 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.784006 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:46.787156 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:46.793176 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:46.820501 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:46.883742 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:46.886884 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:46.888163 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.890647 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:46.895366 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:46.900386 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:46.995959 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.001425 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:47.002245 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:47.006733 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:47.009078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.018615 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:47.018930 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:00:47.034958 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.035089 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:47.035338 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.043758 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.046108 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.064021 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:47.065374 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.069555 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:47.073838 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:47.084228 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:47.087260 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.098632 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.178212 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.179992 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.181842 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.185215 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.186055 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.186168 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:47.202589 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:47.260666 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:00:47.261317 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.283582 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.288681 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.289091 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.289848 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.298007 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.299650 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:47.368139 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.398164 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.409157 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.410259 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.410413 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.411259 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.426761 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.427033 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.434216 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.438058 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.441340 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.445053 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.455474 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:47.475564 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:47.480126 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:47.488698 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:47.496872 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:47.542122 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:47.545405 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.548599 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.549705 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.551033 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.551585 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:47.558099 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:47.572368 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:47.597540 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:00:47.604611 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:47.652371 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.653949 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.656172 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.658394 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.663097 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.669687 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.754070 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.758492 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.758698 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.759690 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.767822 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.769869 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.812451 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.815546 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.818183 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.819931 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.832067 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.849786 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:47.851133 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:47.907735 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:47.912468 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:47.921056 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:47.922188 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:47.922374 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:47.922626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:47.965139 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:48.030377 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:48.124017 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:48.125764 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.130461 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.130980 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.132860 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.140558 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:48.144008 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:48.155009 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:48.200567 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:48.231066 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.234700 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.235177 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.235802 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.238945 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.241055 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:48.257071 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:48.329124 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.332077 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.332784 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.339025 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.340224 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.341499 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.443322 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.443799 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.452718 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.453070 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.460161 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.487429 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.489126 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.489956 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.493749 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.497143 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.501657 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.513886 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:48.527701 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:48.590202 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.591107 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.591734 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.592538 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:48.606297 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:48.607167 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.608448 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:48.673442 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:48.693455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.695645 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.697774 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.699054 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.702103 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.710199 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:48.736469 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:48.757272 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:48.803841 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.809729 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.816841 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.821796 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.825059 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.829061 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:48.847990 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:48.863965 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.869051 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:48.869267 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.873684 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.878993 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.882905 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:48.914288 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:00:48.955550 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:48.958307 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:48.960028 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:48.962695 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.969051 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:48.970069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:48.982710 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:49.063998 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.064546 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.066136 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.069061 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.075021 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.077433 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:49.172339 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:49.182623 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.193918 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.202559 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.202560 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.206645 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.209573 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.218039 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.237229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.238446 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.239797 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.241972 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.242743 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:49.261965 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:49.334352 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:49.345415 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.347537 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.349483 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.355850 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.357873 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.360123 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:49.365518 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:49.437844 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.445353 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.445717 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.449031 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.449363 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.449741 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:49.494288 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:00:49.500392 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:49.550375 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.552196 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.553997 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.568938 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.568955 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.569984 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.571722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.583585 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.584333 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:49.608466 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:49.608567 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:49.609566 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.619086 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.625244 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:49.634177 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:49.679661 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:49.765877 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:49.787403 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:49.789800 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:49.803284 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:49.885885 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.890537 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.893041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.900684 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:49.904521 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:49.905381 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.911139 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:49.913676 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:49.914461 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:49.952653 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:49.954313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:49.956833 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:49.957216 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:49.958789 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:49.959396 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:49.962406 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.049229 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.054730 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.054987 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.059211 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.060057 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.076185 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:50.140012 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:50.146775 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:50.160367 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.162803 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.166239 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.169059 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.171166 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.177362 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:50.178131 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:50.222967 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:50.264048 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.269505 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.269796 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.271497 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.304574 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.308060 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.308138 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.322952 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.332279 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.341482 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.342696 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.346545 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:50.351222 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:50.425794 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.427568 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.429414 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.429500 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.430346 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.439904 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:50.450639 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:50.490430 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:50.527678 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.532724 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.532881 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.538821 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.540535 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.544817 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:50.614006 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:50.644098 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.650344 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.651767 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:50.659812 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:50.668879 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.674782 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.674870 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.675256 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:50.675729 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 20:00:50.686573 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:50.698062 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.698724 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.702533 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.709453 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.714901 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:50.760525 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:50.796321 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:50.799556 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:50.802732 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.805871 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.807605 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.812404 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:50.820040 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.916457 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:50.916869 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.917243 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:50.917629 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:50.918705 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:50.920945 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.023788 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.028506 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:51.036144 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:51.036788 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.037833 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.065379 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.070386 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.075891 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.079499 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.093207 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.093725 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.105551 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.108894 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:51.146981 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:51.186947 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.187602 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.189270 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.191890 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.195149 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.196058 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.287163 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.293206 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.297194 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.300841 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.310961 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.311168 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:51.312271 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:51.345551 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 45 similar messages]
I20240501 20:00:51.388859 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.389097 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.391088 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:51.392607 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:51.394623 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:51.400821 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:51.402380 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.405336 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:51.424714 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:51.425694 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:51.451844 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:00:51.594132 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.594983 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.595893 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.605177 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.620707 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.629914 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:51.648345 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:00:51.696712 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:51.717566 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:51.718063 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.726871 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.732146 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.733373 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.747676 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.751595 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.752180 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.755436 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.757066 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.761268 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.772769 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.773780 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.864488 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.865013 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.868321 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.871747 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.872629 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.883769 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.963516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:51.963560 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:51.964123 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:51.966967 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:51.966998 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:51.967633 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:51.992012 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:52.024597 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:52.082461 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.082564 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.084475 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.093657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.096369 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.103744 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.106133 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.109195 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.129220 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.131652 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.135524 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:52.147294 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.148456 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:52.151698 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:52.155673 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.189867 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:52.197903 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.211455 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:00:52.230677 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:52.233486 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.234865 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.241523 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.247068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.247597 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.247941 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:52.267678 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:52.336292 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.341853 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.342095 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.346621 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.349006 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.365726 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:52.400929 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:00:52.401620 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.430285 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:52.445977 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.452549 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:52.459012 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.459335 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.460041 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:52.462749 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:52.466491 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.471607 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.477172 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.483804 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.490975 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.491868 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.496503 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.504797 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:52.524281 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:52.524497 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.601881 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.602039 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.605060 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.609393 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:52.609920 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:52.613222 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:52.613489 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:00:52.627784 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.707434 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.709623 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.713474 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.721385 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.721414 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.729213 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:52.821094 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:52.821610 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.828984 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.829695 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.829933 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.848357 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.849278 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.849480 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.851886 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.867270 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.872519 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.872764 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.901454 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:52.947242 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:52.957484 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:52.973282 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:52.973841 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:52.976962 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:52.986235 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:52.989593 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:52.992277 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:53.017190 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:53.170419 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:53.176708 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:53.187211 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.187510 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.192426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.192579 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.194754 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.198275 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:53.202394 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:53.290262 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.291563 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.292204 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:53.292377 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:53.294646 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.295346 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.300676 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:53.365178 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:53.390286 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.391631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.394110 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.394498 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.403962 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.422040 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:53.438156 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:53.456532 29173 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:00:53.456626 29173 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:00:53.457109 29173 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:00:53.458165 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:53.458741 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:53.458806 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:00:53.458920 29173 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:00:53.509451 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.514058 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.519793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.520737 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.521689 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.525175 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.530875 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.531466 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.541112 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.544188 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:53.547791 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:53.553709 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.561684 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.658423 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.661226 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.662421 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.663064 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.665478 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.675086 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:53.702343 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:53.756158 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.758414 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:53.758466 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:53.758919 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.762528 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.771059 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.772946 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:53.790894 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:53.871119 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.871742 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.879889 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.881498 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.882898 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:53.883047 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:00:53.899767 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:53.900923 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.908353 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:53.908373 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:53.909561 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:53.911121 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:53.929340 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:53.948482 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.003182 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:54.026053 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.026522 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:54.026773 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:54.027412 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.029281 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.032341 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.055912 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.141609 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.145751 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.146524 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:54.147781 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.148454 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.151441 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.169817 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.255054 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.255656 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.261325 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.263098 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.289906 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.291101 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.308295 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.312461 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.314990 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.316977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.325496 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.335827 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:54.368791 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:54.418365 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.419983 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.420097 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.431681 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.432101 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.438766 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:54.444669 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.513650 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.518533 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:00:54.518625 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.519212 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.520121 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.520905 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.532763 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.537072 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.567755 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:54.628520 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.635656 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.638816 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.639787 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.641052 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.663419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:54.671447 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:54.686854 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.690145 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:54.694690 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.695076 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.703912 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.710299 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.714453 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.727370 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.751770 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:54.817405 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.836740 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.880890 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:54.911392 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:54.956948 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:54.958106 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:54.958751 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:54.966181 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:54.969787 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:54.990036 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:54.990824 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.021649 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:55.035225 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.036866 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.040822 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.043470 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.047967 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.048244 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.113193 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:55.154008 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.158141 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.161092 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.168818 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.169430 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.177188 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:55.185505 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:55.187525 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:55.276145 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.283449 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 20:00:55.287220 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.288085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.289079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.296103 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.302323 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:55.318961 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:55.329370 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.333312 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.344223 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.345599 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.348484 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.369297 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:55.390302 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:55.429250 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.433166 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.435820 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.436589 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.439563 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.455313 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.523808 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:55.526296 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:55.527338 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.530061 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.534181 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.542582 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.550911 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.556413 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:55.644122 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.648711 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.648977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.650799 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.656996 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.663359 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.697650 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:55.710759 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.713986 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.715965 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.716476 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:55.724028 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:55.726262 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.728339 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.792109 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:55.793090 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.796972 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.798902 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.807489 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:55.815588 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:55.815892 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:55.817860 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:55.857033 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 20:00:55.899312 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:55.902685 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:55.903774 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:55.904987 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.905633 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:55.912848 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:00:55.982867 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:56.005100 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.005810 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.013451 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.024502 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.024860 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.027662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.075838 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.085773 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.087056 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.088704 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.101994 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.102960 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.160300 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.164112 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.166689 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.171756 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.172523 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.182674 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:56.245862 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:56.247807 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:56.251562 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:56.260983 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.261454 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.267117 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.271613 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.272243 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.280205 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:56.325353 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:56.334359 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:56.383994 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.385417 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.387595 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.389788 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.393996 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.398550 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:56.437788 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:00:56.442886 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:00:56.447609 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.449040 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.454375 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.461864 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.464229 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.464449 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:56.492923 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:56.627686 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:56.668303 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:56.701681 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.701964 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.705883 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.713670 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.716188 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.721339 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:56.722960 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 20:00:56.810508 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.811655 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.814788 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.814844 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.826738 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.834821 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.837846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.842571 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:56.858917 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:56.866384 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.871634 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.874176 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.883545 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:56.889802 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:00:56.910338 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:56.921280 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:56.947851 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:56.966001 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:56.968218 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:56.973050 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:56.973914 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:56.975095 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:56.977298 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.071053 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:57.073096 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.077383 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.077641 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.080535 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.089844 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.103103 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.168763 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:00:57.177232 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:57.192059 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.196784 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.198606 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.202849 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.206514 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.206768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.207480 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.207931 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.208153 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.221058 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.255650 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.256369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.279228 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:57.294117 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:57.339128 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.340602 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.341233 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.342705 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.343545 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.357705 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.365684 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:57.379420 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:57.411471 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:57.442200 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.445391 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.450078 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.450307 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.452075 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.452296 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.501960 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 20:00:57.502584 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:57.523728 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:57.551586 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.552067 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:57.557781 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:57.558370 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.561084 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.580782 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.583698 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.600580 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.605178 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.606732 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.624966 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.627604 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.634052 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:57.679461 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:57.691848 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:57.716298 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.718250 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.722069 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.727717 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.733011 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.743700 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:57.775743 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:00:57.805953 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:57.813731 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:57.821604 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.829128 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.829145 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.835618 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.838706 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.853588 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.917383 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.919543 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.933815 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.941473 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.944110 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.961156 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:57.967200 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.969031 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:57.977455 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:57.980680 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:57.982467 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:57.989748 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.048086 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:58.090163 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.093627 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.095292 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.096249 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.096586 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.104797 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:58.117110 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.165249 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:58.191682 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:58.278415 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:58.279868 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:58.279968 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.281059 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.281437 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.289072 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.291031 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.301183 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:58.357973 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:58.394928 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:58.398488 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.398692 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.398914 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.403237 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.403622 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.403726 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.463762 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:58.493048 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.496822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.497208 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.498629 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.517321 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.526548 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.598251 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.601445 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.608757 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.610453 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.622660 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.632759 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.660921 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:58.666244 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.668737 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.673914 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.681790 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.682286 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.690757 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:58.766712 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:58.769531 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.773906 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.774590 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.786574 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.786820 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.795435 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.869025 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:58.873214 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.877132 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.878649 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.891202 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:58.892169 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:58.966465 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:58.969444 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:58.977927 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:00:58.993739 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:58.994678 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.001104 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.001881 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.013437 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.022331 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:59.027787 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.030982 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.057979 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.059924 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:59.094249 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:00:59.141911 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.144078 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.144476 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.147130 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:59.150331 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.156875 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:59.173439 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:00:59.197853 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:59.232286 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:00:59.246462 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:59.248972 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:59.249473 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.250926 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.251279 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.254765 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.266793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:00:59.286674 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:00:59.358083 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.359642 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:59.361310 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:00:59.364140 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:59.365443 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.368381 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.369580 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.395195 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.418813 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.425287 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.427331 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.430670 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.431341 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:59.432710 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.521313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.521888 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.524514 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.527760 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.533852 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.534574 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:59.535490 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:00:59.613524 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.617774 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:00:59.625005 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.627905 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.629730 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:59.641618 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:59.644513 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.714887 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.717164 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.725592 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.727695 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.732793 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:00:59.735532 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:59.771091 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:00:59.775223 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:00:59.776162 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:00:59.779747 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:00:59.782346 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:00:59.783751 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.784262 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:00:59.787391 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:00:59.828568 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:00:59.830579 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:00:59.889552 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:00:59.929579 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:00.023190 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:00.036578 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.036886 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:00.041780 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:00.042061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.045727 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.057555 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.060111 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:00.072198 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:00.087186 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:00.127137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.131134 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.131664 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.134788 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.142091 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.147488 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:00.200125 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:00.211941 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:00.231106 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.232429 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.234642 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.238634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.246510 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.252856 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:00.277473 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:00.340296 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:01:00.341634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.348383 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.349941 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.351377 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.355084 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.367937 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:00.380455 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:00.401293 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.401479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.402118 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.407306 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.419950 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.429955 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.503252 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.504030 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.506817 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.507087 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.509239 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.525600 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:00.536826 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:00.598270 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.601790 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.605033 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.605237 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.606629 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.619302 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:00.649108 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:00.726866 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.727537 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.734573 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.735348 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:00.739041 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:00.739651 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.741310 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.742583 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.760710 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.761020 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.767031 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.790699 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.795866 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:00.802704 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:00.856384 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:00.857147 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:00.859278 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:00.868644 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.868885 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:00.871127 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.876561 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.876931 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.881417 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:00.909276 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:00.925310 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 20:01:00.971159 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:00.973212 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:00.977731 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.982559 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:00.982785 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:00.985440 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:01.089474 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:01.094547 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.097091 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.115888 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.116034 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.116307 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.116967 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.125910 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.127875 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.130730 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.131191 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:01.142537 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:01.158202 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.168531 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.252858 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.253372 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.253962 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.259033 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.267024 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.282900 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.355901 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.356755 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.358659 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.363207 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.369874 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.390764 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:01.471920 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:01:01.473146 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:01.476305 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:01.483958 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.485189 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.487871 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.488662 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.494290 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:01.495200 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:01:01.502286 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.505144 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.513689 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.517208 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.522068 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.529094 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.550652 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:01.634794 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:01.645335 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:01.690569 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:01.794327 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:01:01.827983 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:01:01.837070 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.843751 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.849486 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:01.855036 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:01.855470 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:01.856154 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:01.864984 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.885869 11092 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.886633 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.887887 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.891511 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:01.894006 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:01.900563 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.906773 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.923178 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.987664 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:01.988135 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:01.989459 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:01.989805 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:01.991694 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:01.993996 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:02.030704 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:02.090689 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.095206 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.096223 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.096882 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.099336 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:02.099396 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.101339 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:02.118083 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:02.121258 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.219658 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:02.220161 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.223623 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.230777 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.242471 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:02.243038 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.244776 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.245860 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.257050 10963 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.257184 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.257246 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.257870 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.258733 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.262050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.378837 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.380389 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.384907 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.391003 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.391798 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.394321 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:02.460662 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:02.476346 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.477689 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.479542 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.485201 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.490134 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.492202 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:02.497499 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:02.505201 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:02.543813 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:01:02.587745 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:02.591347 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.592144 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.594152 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.598013 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.603045 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.610862 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.612210 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:02.620004 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:02.624404 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:02.634774 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.648241 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.649217 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.649927 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.651110 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.664882 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.749347 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.753067 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.755129 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.755627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.766947 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.768994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:02.780998 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:02.856369 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.861488 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.862653 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.865211 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.865212 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.866904 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:02.886137 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:01:02.931988 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:02.948940 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:02.965086 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.974054 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:02.975435 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:02.976802 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:02.982750 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:02.995904 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:02.996120 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.000409 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.004050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:03.016484 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:03.017670 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.018090 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.018303 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:03.045313 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:03.117679 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:03.117720 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.120208 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.122189 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.125510 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.129043 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.129770 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.218866 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.221915 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.223850 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.230830 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.235733 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:03.236047 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:03.237519 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:03.237533 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.324369 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.325659 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.326344 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.334723 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.339051 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.341087 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:03.405933 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:01:03.459560 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:03.484510 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:03.528990 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.531412 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.533715 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.534533 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.535336 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.547593 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:03.554282 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:03.652599 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.653898 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.656212 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.663012 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:03.665966 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:03.669064 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.669998 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.677803 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.685662 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.693329 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.696790 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.706843 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.710315 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.793530 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.798416 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.798964 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.800163 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.812805 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.812961 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.898641 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:03.900173 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:03.904318 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:03.911010 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:03.913769 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:03.917061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.000448 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:04.004976 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:04.006187 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.013260 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.015895 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.016028 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.018098 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:04.035002 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:04.043036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.047992 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.051342 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.052676 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.058679 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.059787 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:04.106222 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.162604 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.165421 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.166190 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.167603 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.173420 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.179669 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:04.194448 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:04.227548 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:01:04.266553 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.269824 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.270849 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.271117 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.271587 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.283762 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:04.333653 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:04.336067 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:01:04.337646 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:01:04.339749 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:04.377715 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:01:04.385934 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:04.386904 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.388713 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.389192 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.395000 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.401875 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.407047 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.415725 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.421933 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:04.424479 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.427453 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:04.444506 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.447660 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.447839 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.449488 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:04.472846 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.527016 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.530331 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.532739 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.539997 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.545940 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.545939 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:04.592039 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.637501 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.642303 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.642585 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.644923 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.645381 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:04.646220 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:04.648175 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:04.750597 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:04.754783 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.755522 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.756114 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.757026 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.769686 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.781374 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.792137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:04.793949 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:04.815013 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:04.815697 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.817708 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.820580 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.823216 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.899140 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:04.900066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:04.907456 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.913422 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:04.914414 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:04.925959 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:04.961547 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:05.003528 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:01:05.034930 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:05.121063 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.122295 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.130385 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.131219 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.132831 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.136972 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:05.159081 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:05.173871 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:05.179630 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:05.222307 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:05.237354 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.240983 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.244189 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.245047 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.252041 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:05.262290 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:05.269441 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:05.309166 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:05.341284 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:05.345786 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:05.347159 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.348331 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.353071 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.353725 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.364284 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.460809 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.465754 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.470364 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.481003 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.481185 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.481285 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.484644 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.496665 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.503650 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.506553 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.509841 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.527976 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:05.574613 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:05.609382 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.610472 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.619896 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.627835 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.630220 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.630851 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.716162 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:05.719355 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:01:05.720047 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.722090 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.727715 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.741405 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.741765 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:05.761884 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:05.790230 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:05.819651 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.825584 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.825659 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.827348 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.843210 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.846629 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.889851 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:05.897578 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:05.897655 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:05.898432 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.900082 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:05.901857 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:05.943459 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:05.967368 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 84 similar messages]
I20240501 20:01:05.998956 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:05.999439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.002398 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.008410 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.022065 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:06.025137 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:06.027958 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.119922 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.121754 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:06.121976 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:06.126861 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.134150 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.138424 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.161617 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.193564 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.199317 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.200785 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.202245 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.206079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.214200 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:06.245829 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:06.289896 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.289999 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.294579 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.294958 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.301154 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.305191 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:06.321362 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:06.390817 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:06.399087 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.401933 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.403297 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.405562 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.409353 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.427122 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:06.485126 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:06.503794 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:01:06.519981 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.520370 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.526490 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.529945 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.532680 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.532763 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.534663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:06.551717 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:06.556800 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:06.557637 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.557734 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.572203 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:06.575117 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:06.577752 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.584304 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:06.661006 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:06.661039 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.666833 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.670032 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.677166 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.685174 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.689934 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:06.694450 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:06.703954 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:06.719399 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:06.821420 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:06.857206 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:06.858683 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:01:06.860595 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.863533 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.863870 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.864351 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.865618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:06.882079 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:06.914876 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:01:06.968992 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:06.969977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:06.970220 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:06.971437 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:06.983247 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:06.989557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:06.999042 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:01:07.043519 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.069020 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.070427 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.070749 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.072999 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.081293 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.092053 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.098719 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.156278 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:07.175177 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.177681 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.185432 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.190737 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:07.192072 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.193097 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.195931 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:07.217381 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:07.234328 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.241086 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.247272 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.248757 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.249205 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.250650 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:07.322324 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.338701 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.338733 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.346622 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.347834 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.349048 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.350297 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.446231 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.446470 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.448872 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.449540 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.457440 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.461251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.493479 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:07.507542 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.549608 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.556208 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.560631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.561407 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.563789 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.566712 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.597545 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.600219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.603147 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.603698 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.617785 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.619015 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.626310 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:01:07.651496 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:07.711246 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.714042 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.719344 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.723606 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.726907 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.727159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:07.737279 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:07.782040 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.811020 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.816119 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.817788 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.822037 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.838642 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.841871 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:07.862946 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.913434 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:07.914036 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:07.915179 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:07.917735 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.922716 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.932827 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:07.932981 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:01:07.933657 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:07.956808 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.975615 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.976365 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:07.980129 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:07.981997 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:07.994371 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:07.999074 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:08.030172 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:08.079932 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.081727 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.087519 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.095394 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.095615 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.101884 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:08.133623 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:08.149960 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:08.183405 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.186997 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.189417 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.190730 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.191438 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.201212 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:08.229238 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:08.283140 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.283618 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.288486 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.288964 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.289410 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.290539 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.298949 4375 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:01:08.299036 4375 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:01:08.299371 4375 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:01:08.301079 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:08.301189 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:08.301226 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:01:08.301393 4375 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:08.348453 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.350411 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.352320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.354573 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.354878 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.380931 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:08.453495 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:08.456750 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:08.501865 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:08.599440 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.601518 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.602591 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.607230 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.607935 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.609272 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.698236 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.698516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.701041 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.702000 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.702522 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.709656 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:08.758319 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:08.763971 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:08.795025 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.801004 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.802224 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.803817 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.806696 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.809096 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:08.844995 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:08.905872 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.906816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.917474 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.918171 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:08.926709 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.927973 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:08.946003 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:08.970913 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:08.976006 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.978809 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:08.985487 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:08.993479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:08.998437 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:09.046608 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:09.050902 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:09.060096 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.063920 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.067970 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.073832 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.075343 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.096861 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:09.131362 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:09.135979 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:01:09.136534 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:09.170266 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.171618 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.172798 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.175384 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.178050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.208418 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:09.243296 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:09.248821 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:09.292361 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:09.297842 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:09.305800 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:09.306874 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.309525 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.309897 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.311532 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:09.328693 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:09.332510 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.332927 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.333905 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.340199 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.348733 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.352155 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.366084 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:09.371785 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:09.436442 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:09.451910 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.452843 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:09.458842 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:09.461598 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.463161 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.463935 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.469180 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.552707 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.557031 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.561038 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.562512 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.569983 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.581369 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:09.621327 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:01:09.673333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.683068 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:09.684715 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:09.689121 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:09.696033 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:09.701103 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.701903 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:09.703584 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:09.705040 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.707770 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.717260 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.717279 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.717866 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.720423 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.725708 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:09.772794 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:09.839874 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.840883 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.841765 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.855626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.857872 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.858788 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:09.923830 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:09.944545 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:09.945405 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:09.952059 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:09.952852 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:09.953845 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:09.969986 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:10.062027 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:10.065248 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.066112 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.069654 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.075251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.076644 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.085500 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.115556 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.117256 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.122783 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.130311 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.131369 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.138710 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:10.298529 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:10.309334 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:10.387646 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.390477 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.391155 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.399471 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.407856 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.415719 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.457655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.460737 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.465086 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.466486 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.467736 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.472271 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:10.474902 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:10.531349 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:10.550274 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:10.558439 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.559011 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.561987 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.564867 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.567082 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.580407 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:10.600617 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:10.670701 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.673434 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:10.674347 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:10.676929 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.678449 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.684669 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.691207 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:10.712550 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:10.740430 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240501 20:01:10.753759 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.759370 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.759464 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.759903 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.763557 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.781086 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:10.796379 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:10.822281 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:10.829183 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:10.830504 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.830951 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.836412 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.838579 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.862349 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.933444 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:10.934918 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:10.938796 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:10.945047 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.945397 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:10.946421 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:11.010627 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:11.043606 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:01:11.061872 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.062050 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.062641 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.062714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.063406 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.078996 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:11.128513 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.131067 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.134567 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.139183 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.141619 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.143999 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:11.151822 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.169606 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.213201 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:11.218837 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.219461 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.219695 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.223132 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.229074 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:11.237404 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.320827 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.323108 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.323541 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.329107 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:11.332513 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.333127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:11.431996 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:11.433246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.439065 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.441007 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.445060 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:11.451808 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.480479 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.488493 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:11.492981 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:11.494215 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.498093 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.500190 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.509306 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.516793 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:11.522898 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:01:11.535923 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.554380 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:11.589718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.593631 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.596647 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.597236 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:11.602113 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:11.613746 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:11.622213 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.622355 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:11.664242 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:11.699250 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.700745 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.706549 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:11.707911 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.714404 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.715216 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:11.725704 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.733265 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:11.816273 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.816833 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.819633 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.827296 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.831017 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:11.833179 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:11.834578 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:11.837553 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:11.851186 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:11.858595 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.869742 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.872296 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:11.885416 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:11.919348 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.926944 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:11.967211 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:01:12.002008 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:12.005939 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:12.071404 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:01:12.071996 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:12.085563 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:01:12.178637 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.186787 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.189371 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.192595 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.192716 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.209821 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.216670 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.217599 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.222479 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.235954 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.236470 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.238914 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:12.247855 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:12.278976 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:12.284480 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:12.328095 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.328876 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.332132 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.333539 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.336274 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.337667 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:12.362202 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:12.440866 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.443151 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.447127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.451357 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.457345 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.466315 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:12.512696 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:12.548981 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.550117 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.553160 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.553884 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.557384 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.561710 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:12.598603 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:12.599745 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:12.609352 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.613454 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.618079 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.625063 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.625890 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.633602 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.712118 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.712365 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.714218 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.717335 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.723413 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.739248 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:12.746068 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 20:01:12.818377 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.822309 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.827397 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.829371 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.842293 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.849040 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.906558 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.908104 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.910737 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.917928 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.932554 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:12.932802 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.978735 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:12.984134 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:12.986553 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:12.987188 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:12.992308 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.005344 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:13.025352 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:13.031378 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:13.046587 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:13.074304 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:13.075820 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:13.092550 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.092883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.094038 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.097690 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.099691 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:13.107235 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:13.180596 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:13.217507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.224581 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.227965 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.229135 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:13.233359 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.253361 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.255699 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:13.263316 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:13.266562 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.267784 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.270401 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:13.270687 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:13.272455 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.299692 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:13.377312 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:13.378360 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.380781 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.387521 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.391623 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.394474 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.474237 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.480598 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.481966 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.482640 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.483063 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:13.491065 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:13.499846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:13.551987 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:13.578821 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:13.598660 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.600165 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.604328 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.607214 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:13.618510 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:13.627285 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:13.628890 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:13.629463 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.631551 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:13.638576 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:13.641428 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.644150 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.644850 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.663929 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.671089 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:13.933319 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:13.962054 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:13.968387 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:13.975498 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.977943 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.981693 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:13.984527 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.992988 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:13.994732 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.005365 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.005998 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:14.013959 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:14.033071 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.034252 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.113009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.115857 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.118748 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.120394 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.120594 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:14.127631 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:14.135941 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:14.136407 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:14.195854 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:14.215734 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:14.219537 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.224032 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.224259 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.226576 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.226749 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:14.237697 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:14.243532 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:14.259013 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:14.315913 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:14.340046 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.340988 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.344043 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.345860 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.349637 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.360437 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.361335 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.368563 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.378633 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.381139 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.397020 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.410861 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:14.423578 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:01:14.442131 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:14.488960 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.489085 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.491768 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.491768 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.492182 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.513907 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:14.554653 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:14.565536 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:14.577548 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:01:14.592522 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.593565 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.594379 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.597339 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.598002 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:14.627405 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:14.629877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.712756 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.715853 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.725255 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.728829 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.731307 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.733023 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.741464 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:14.741755 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:14.764227 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.764614 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.765305 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.772928 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:14.777190 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:14.790174 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:14.820691 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:14.832523 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:14.875032 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.882985 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.884259 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.886797 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.887180 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.896363 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.984751 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:14.985154 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:14.986642 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:14.987139 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:14.990103 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:14.991276 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.057554 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:15.095067 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.096328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.101107 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.105347 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.123978 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.133224 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.143460 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.144229 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:15.151697 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.155908 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.162380 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.165338 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.185544 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.271698 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.273219 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.275509 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.283560 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.284425 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.289645 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:15.301077 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.309048 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:15.365376 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:15.383893 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.384469 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.390136 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:15.394116 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:15.399717 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.399782 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.400883 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.479362 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.480407 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.482374 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.484880 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.494076 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.502772 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.511114 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:15.514349 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:15.671514 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:15.692334 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.694607 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.697537 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.698022 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.701128 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.706451 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.798517 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.801218 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.804047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.807124 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.808913 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.812716 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.827896 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.833909 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:15.847523 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:15.847875 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.858518 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.862875 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.879645 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:15.886716 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:15.894928 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:15.910017 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:15.945493 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:15.955539 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:15.960223 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:15.961969 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:15.962559 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:15.971525 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:15.973145 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.011368 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:16.072250 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.075420 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.078385 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.081653 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.093055 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:16.105818 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:01:16.118214 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.139637 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:01:16.165274 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.169704 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.170737 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.176091 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.185220 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.185860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.237761 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.238606 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.243083 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.246353 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.246994 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.248682 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:16.255323 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.284476 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:16.335145 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.338575 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.339550 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.342638 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.363447 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.376008 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.462320 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.463730 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.468992 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.469099 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.474771 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.478091 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.516865 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:16.530130 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.533092 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.533391 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.534763 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.535452 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.550123 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:16.553252 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:16.564226 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:16.584986 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:16.616519 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.620791 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.621187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.627965 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.638258 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.646616 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.657421 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:01:16.722363 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.725625 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.728945 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.731256 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.731254 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.739363 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:16.761914 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:16.770874 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:16.786542 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:16.829691 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:16.837627 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.840377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.846127 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.848209 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.849471 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.850646 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.905289 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.907132 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.912570 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.915418 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.918699 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:16.922693 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:16.929803 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:16.955359 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:16.979609 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:16.980640 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:16.982648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:16.983160 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:16.989059 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:16.998888 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:17.069157 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:17.086503 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:17.087983 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.091547 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.093071 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.094724 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.095091 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.115916 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:17.122478 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.193823 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:01:17.200374 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:01:17.223053 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.224232 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.224885 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.234345 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.236260 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.247581 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:17.255383 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:17.328230 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.362740 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:01:17.367194 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.388229 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:17.402683 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.404771 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.409209 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.414506 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.414731 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.429831 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:17.437445 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.448944 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:17.510361 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.512100 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.513414 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.525233 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.525295 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.547425 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.582634 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.584614 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.584779 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.585289 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.588747 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.590788 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.673831 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.678860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.681259 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.682395 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.689263 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.697219 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.771087 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.776029 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.777233 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.779474 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.780393 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.805791 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:17.813863 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.817520 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:01:17.833256 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:17.895437 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.895458 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.896080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.896335 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:17.906833 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:01:17.920382 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:17.922797 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.927201 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.954121 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:17.956822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:17.958732 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:17.962155 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:17.982000 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:17.985448 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.039474 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.041743 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.042310 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.049360 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.049930 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.050273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:18.085733 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:18.151036 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:18.152163 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:18.152860 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:18.154574 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.155328 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.155592 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.169731 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.188072 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.277107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.282964 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.283962 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.292096 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.295142 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.295186 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:18.297398 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:18.322649 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:18.322784 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.322870 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.330186 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.335881 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.338888 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.342092 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:18.361564 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:18.432555 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:18.435047 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:18.436604 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:18.440603 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.441530 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.444407 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.449707 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.455461 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:18.476179 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:18.497154 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:18.541664 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.543633 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.546499 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.547659 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.550558 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.572127 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.657928 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.664680 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.667908 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.673357 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.675690 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.686178 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:18.705868 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:18.712865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.718708 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.719983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.720227 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.720981 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.726052 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.817586 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.817999 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:18.821115 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:18.824385 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:18.834241 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:18.836133 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.840173 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:18.848253 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:18.908681 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:18.978595 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:18.979480 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:18.981205 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.990964 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:18.993731 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.008270 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:19.015424 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.022817 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.108572 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:19.130466 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.131229 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.133724 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.136274 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.137634 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.162775 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.244429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.244851 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.247581 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.254355 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.254462 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:19.269328 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:19.271704 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:19.297914 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.306456 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:19.328320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.329798 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.330358 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.331101 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.339366 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.353829 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:19.394261 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.398242 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:19.409027 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.409202 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.413116 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.415511 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.422847 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.436549 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:19.445962 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:19.513600 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.517931 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.518437 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.518570 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.522842 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.533515 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:19.552816 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:19.564777 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.589887 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:19.621820 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.623271 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.624171 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.633965 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.634852 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.642886 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.704380 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.704595 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.716310 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.716307 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.726990 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.734009 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:19.762712 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:19.779047 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.779430 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.779678 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.781064 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.782009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.785208 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:19.788141 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.805652 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:19.813812 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:01:19.882565 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.882926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.883435 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:19.884246 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.893662 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:19.909034 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:19.969107 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:19.973712 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:19.992852 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:19.993958 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:19.995829 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:19.998353 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.009945 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.013253 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.068835 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.069293 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.072943 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.073719 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.080526 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.081523 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:20.106763 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:20.156592 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.160583 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.161258 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.161753 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.168046 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.173382 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:20.219175 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:20.264856 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.270169 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.271131 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.271384 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.285671 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.285780 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:20.346159 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:20.368106 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:20.374948 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.378042 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.378659 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.385782 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.402086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.405383 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.427201 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.444010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.444773 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.445971 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.452090 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.453413 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.525775 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:20.526472 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:20.529496 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.529626 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.532384 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.540135 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.549605 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:20.663596 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:20.696437 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:20.702805 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:20.738612 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:20.745930 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.749505 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.749637 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.750424 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.750682 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.751026 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:20.789345 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:20.844552 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.847991 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:20.848239 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.848738 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.863124 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.863584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.939579 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:20.944486 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:20.944983 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:20.946185 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:20.948253 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:20.953884 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:20.972429 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:20.989725 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:21.073596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.076429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.084120 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.094604 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.096132 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:21.098492 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:21.098691 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.105111 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.113822 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.116782 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.118595 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.135959 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.152415 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.160962 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:21.181218 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 20:01:21.181758 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:21.231950 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.234661 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.236331 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.236428 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.238865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.248348 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.278662 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:21.296130 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:21.331032 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.332898 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.333477 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.334031 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.335458 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.341816 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.448591 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.450131 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.450896 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.454694 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.460408 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.468194 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.468698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.469244 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.470016 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.475309 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.483748 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:21.484205 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:21.491071 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:21.491636 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.506464 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:21.515342 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.592188 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.596246 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.597476 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.597941 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.601648 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.604184 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.638777 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.661409 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 46 similar messages]
I20240501 20:01:21.704295 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.704301 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.705281 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.710572 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.711359 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.713812 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.765870 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.780974 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.800694 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:21.810601 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.825706 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.825840 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.831425 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.839676 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.841892 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:21.846956 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:21.848161 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.855329 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:21.856599 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:01:21.868732 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:21.869700 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.870347 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.872160 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.877597 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.881088 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:21.893466 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:21.922569 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:21.968111 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:21.971590 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.980619 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:21.981305 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:21.981344 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:21.992272 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.073642 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.074575 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.083343 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.089115 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.095381 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.095615 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.174187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.177136 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.177644 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.179136 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.189049 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.217897 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:22.224282 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:22.227046 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:01:22.236340 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.241688 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.243047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:22.243361 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:22.245543 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.246817 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.264628 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:22.384541 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:22.387696 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:22.448320 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:22.479431 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:01:22.516232 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.517086 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.518985 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.521101 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.525797 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.529043 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:22.556818 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:22.592070 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.592998 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.594781 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.595198 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.596526 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.605286 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.694260 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.694517 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.700057 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.702253 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.715160 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.727336 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.816452 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.819782 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.822358 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.824741 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:22.830155 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:22.835662 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:22.846493 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:22.851586 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.878907 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.882037 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.890409 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.894572 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.897081 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:22.898362 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:22.935551 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:22.981067 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:22.983515 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:22.983939 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:22.987252 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:22.990264 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.009685 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:23.049121 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:01:23.055979 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:23.099406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.100965 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.101732 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.105477 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.105682 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.117442 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:23.131214 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:23.155604 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:23.207157 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.210124 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.210361 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.212376 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.212795 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:23.245250 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:01:23.247136 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:23.249531 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:23.250921 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.262768 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.265586 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.277596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:23.285917 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:23.289657 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.290269 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.303825 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:23.312681 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:23.390308 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.392822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.394466 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.395965 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.398674 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:23.406248 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:23.410573 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:23.410701 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:23.440784 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:23.495157 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.497541 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.499833 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.501255 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.505259 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.505311 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.585079 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.586963 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:23.587668 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:23.588721 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.595598 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.596596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.613097 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.662849 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.665230 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.668547 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.669222 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.673585 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.680794 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:23.701964 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:23.711053 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:23.738624 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:23.769292 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.769513 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.770938 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.771976 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.774231 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.774644 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.865537 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.868582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.873389 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.879274 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.881726 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:23.892133 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:23.904672 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.940388 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:23.944701 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:23.945544 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:23.951161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:23.953719 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:23.962033 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.008883 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:24.027554 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.028499 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.029510 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.030999 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.039814 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.056041 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.081796 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:24.195019 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:24.244962 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.245085 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.246999 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.255570 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.256028 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.256995 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.291703 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:01:24.311192 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:24.338507 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:24.353425 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:24.359263 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.360273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.360366 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.361106 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.367681 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.382326 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.438314 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:24.462736 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.466254 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.467301 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.467867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.479297 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.480831 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.528035 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:24.547904 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:24.562582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.569411 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.575407 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.575536 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.593044 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.595851 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:24.607494 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:24.618254 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.634274 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.635560 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.636226 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.650808 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.654718 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.732964 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.735637 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.740196 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.745066 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.745389 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.755600 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:24.761847 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:24.773278 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:24.792611 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:24.832548 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.834705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.836239 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.849484 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.850512 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.850934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.924036 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:01:24.946336 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.953085 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:24.954721 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:24.955309 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:24.963091 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:24.976576 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.984467 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:24.987695 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:24.991029 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:24.991254 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:24.996471 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:24.996582 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.008497 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.011587 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:25.017900 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:25.107905 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.108481 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.110615 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.110689 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.120576 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.126271 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.219086 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.219288 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.220562 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.220935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.229597 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.238149 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.319533 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:25.359313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.361701 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.362025 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.365880 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.368538 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.392721 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.397441 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.402990 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.415427 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.417476 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.423730 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.433537 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.447798 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:25.472254 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:25.525336 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.526019 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.527540 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.529328 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.531061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.533097 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.541153 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:25.620110 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:25.624390 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.625236 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.626546 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.633814 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.634186 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.634480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.667393 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:25.668807 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:25.674834 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:25.698666 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:01:25.702280 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:25.743386 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.751761 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.753932 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.754680 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.762761 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:25.764691 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.772832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:25.776721 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:25.787724 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:25.789496 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:25.802636 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.811425 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:25.820497 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:25.957476 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:26.027247 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:26.052073 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:26.112936 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.115988 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.116361 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.120070 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.129817 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:26.143592 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:26.143764 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.147774 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.169884 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.173039 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.177163 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.184252 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:26.190375 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 20:01:26.191696 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:26.199465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.275238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.275547 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.279426 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.279640 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.282045 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.290755 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.297375 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:26.383728 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:26.384585 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.385313 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.385594 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.387341 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.388356 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.399426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.424824 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:26.484059 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.488238 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.491062 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.492802 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.496105 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.524169 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.529263 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.531463 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.533030 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.541461 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.557956 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.569196 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.584020 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:26.609869 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:26.638752 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:26.647701 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.652207 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.655481 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.657930 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.658807 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.675072 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:01:26.678601 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.744565 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:26.756853 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.757386 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.758662 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.759847 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.779100 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.780093 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:26.865454 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:26.867905 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:26.874784 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:26.876710 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:26.877576 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.891206 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.892441 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.893560 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.921133 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:26.922629 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:26.928660 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:26.929785 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:26.930650 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:26.937080 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:26.941627 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:26.957638 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:26.965023 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:27.029629 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.031056 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.033814 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.036345 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.037174 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.037185 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:27.100023 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:27.129617 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:27.142264 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.143031 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.147163 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.148736 12304 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:01:27.148834 12304 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:01:27.149242 12304 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:01:27.150629 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:27.150839 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:27.150915 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:01:27.151135 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.155316 12304 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:27.156674 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.187778 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:27.224655 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:27.237140 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.237808 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.241597 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.241823 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.249688 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.249889 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:27.293824 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:27.306903 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:01:27.307518 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:01:27.316694 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.320014 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.320154 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.329099 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.333213 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.338907 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:27.385111 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:27.419533 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.420276 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.425877 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.426159 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.426545 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.435245 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:27.492895 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:27.493240 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:27.533353 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.533519 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.535022 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.537081 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.547560 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.550555 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:27.584380 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:27.597951 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.600747 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.601622 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.605999 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.609436 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.632594 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:27.748626 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:27.831382 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:27.841624 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.844626 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.844832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.846414 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.846530 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:27.848418 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.951772 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.951809 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.953372 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:27.975351 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:27.978657 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:27.981391 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:27.987290 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:27.995769 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:27.997714 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.000576 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.000659 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.002511 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.002945 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:28.013552 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:28.104166 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:28.110899 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.113021 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.118780 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.118779 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.127204 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.130759 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.142891 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 20:01:28.146046 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:28.191713 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:28.216995 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.217350 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.220014 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.223285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.223522 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.228106 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:28.240650 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:28.253460 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:28.324467 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:28.325824 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:28.327750 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:28.329365 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.332441 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.337661 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.340557 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.350652 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:28.356058 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.359316 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.369367 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.372241 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.386291 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.397394 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.397645 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:28.428505 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:28.490024 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.490128 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.492834 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.496276 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:28.503309 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:28.503968 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.517848 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.576115 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:28.597411 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.599351 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.600200 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.604700 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.607278 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.619874 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.662166 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:28.715490 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.719484 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.721516 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.727623 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.737061 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.737103 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:28.749459 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:28.762326 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.762311 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.767539 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.770635 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.771458 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.784102 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.873584 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.874646 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.875494 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.883628 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:28.888206 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.899138 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.927969 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:28.975667 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:28.980652 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:28.983533 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:28.986706 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:28.988225 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:28.988466 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:28.996299 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:28.996871 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:29.040540 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:29.106254 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.107261 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.108304 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.117244 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.118633 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.118897 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.128724 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.135104 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.145131 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.162420 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.167289 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.169013 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:29.184170 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:29.232761 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:29.262143 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.266513 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.269070 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.284168 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.284478 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:29.290011 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:29.297459 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:29.344687 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:29.439829 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:29.476022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.479351 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.482609 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.485072 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.486430 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.492566 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:29.525871 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:29.569425 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:29.577345 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.582603 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.584358 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.589804 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.592414 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.594561 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:29.614307 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:01:29.630417 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:29.658123 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:29.687634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.689455 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.693274 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.695926 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.701043 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.707167 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:29.789844 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:01:29.809284 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.811424 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.813623 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.818106 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.819790 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.827581 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.837849 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.840751 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.859704 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.866845 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:29.869266 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.883893 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:29.900609 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:29.952387 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:29.971230 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:29.971447 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:29.972642 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:29.974886 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.974967 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:29.995445 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:30.029820 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:30.079106 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.086643 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.088689 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.090260 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.090646 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.092623 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:30.107342 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:30.120678 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:30.182840 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:30.187379 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.188478 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.189324 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.190333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.213501 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.213501 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.232877 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.234102 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.235975 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.252480 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.252718 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.266048 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:30.329447 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:30.348273 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.356128 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.356528 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.358064 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.367017 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.369031 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:30.452728 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:30.462275 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.465797 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.466804 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.469233 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.469956 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:30.471345 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:30.481385 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:30.512427 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:30.534682 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:30.564600 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.567694 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.574113 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.577512 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.588637 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.598393 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.616953 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.630095 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.630113 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.640383 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.648882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.651235 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.735594 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.737057 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.743470 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:30.744104 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:30.746282 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:30.747922 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.750921 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:30.761428 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.838822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.843022 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.847941 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.849709 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.853543 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.858546 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:30.870234 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:30.898617 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:30.946974 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:30.954372 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.955397 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:30.958665 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:30.959740 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:30.961870 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:30.997187 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:01:31.001061 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:31.011775 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.012773 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.013062 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.013614 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.019878 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.020103 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:31.044240 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:31.097604 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:31.108727 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:31.166283 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:31.233342 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:01:31.268352 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.271344 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.272117 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.273860 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.275874 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.297070 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.358000 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.363211 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.363432 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.365006 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.367154 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.387326 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.469375 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.472971 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.473138 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.473191 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.483058 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.486039 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:31.496289 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:01:31.548411 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:31.591194 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:31.595914 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:31.598304 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.601117 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.601272 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.616703 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.629349 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.640103 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.640269 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.640868 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.649051 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:31.652249 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:31.660526 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.663615 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:31.692420 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:31.701606 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:31.752993 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:31.753638 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:01:31.754678 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.755764 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.755892 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:31.757603 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:31.762966 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.763772 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.777485 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.851789 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.856550 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.859869 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.861800 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.866482 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:31.870985 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.968822 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.969988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:31.978979 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:31.985090 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:31.985924 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:31.987473 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:01:31.988811 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:31.993166 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.000133 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.001871 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.011189 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.020006 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.020215 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:32.073626 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:01:32.082280 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:32.109005 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:32.130772 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.134549 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.134979 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.139575 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.140707 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.143597 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:32.148108 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:32.160631 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:32.239492 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:32.240787 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:32.242166 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.244014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.244891 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.249858 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.251259 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:32.291535 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:32.344805 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.350000 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.351727 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.356858 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.357997 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.361070 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.398707 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.406507 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.406507 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.410483 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.426371 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:32.429054 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:32.432477 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:32.472155 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:32.505059 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:01:32.517347 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.519939 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.521050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.529424 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.529767 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.531051 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:32.537117 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:01:32.564306 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:32.578212 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:32.614964 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:32.621912 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:32.622331 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.622488 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.632230 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.647656 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.651067 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.727035 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.735190 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.735874 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.738488 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.749200 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.750510 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.781760 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:32.787005 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:32.787145 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:32.792963 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:32.811450 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:32.819485 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:32.872362 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:32.953800 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:32.961998 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:33.035064 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:33.099694 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.102569 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.104882 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.113590 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.116389 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.127893 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.165778 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.165995 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.168614 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.172950 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:33.175622 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:33.182329 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:33.191565 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.194564 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:33.230243 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 20:01:33.257191 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:33.266041 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.267486 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.268306 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.269187 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.282532 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.283383 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.372664 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.373015 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:33.373535 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:33.377677 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.379101 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.385120 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:33.400285 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:33.400876 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.477928 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.478240 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.478557 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.479970 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:33.482600 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:33.485183 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.487466 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:33.530508 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:33.535516 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:01:33.539530 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.543716 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.544198 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.544960 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:33.545229 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:33.546599 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.548774 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:33.554435 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:01:33.609797 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:33.648447 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.648597 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.648772 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.654086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.659967 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.661588 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:33.682390 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:33.756299 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.759622 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.759935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.775085 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.777073 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.779104 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.836552 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.843741 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.845853 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.850600 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.865137 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.865752 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.923857 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:33.924352 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:33.925820 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:33.927248 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:33.927923 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:33.931702 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.012861 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:01:34.021687 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:34.030057 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.031117 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.031267 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.032225 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.034082 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.034379 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:34.049113 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.050165 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.106827 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:01:34.152052 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.152431 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.153134 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.167691 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.168342 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.168587 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.168787 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.212167 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.222247 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:34.222596 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:34.223876 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.224232 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.226151 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.241351 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.255374 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.305505 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.308221 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.313345 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.314244 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.323942 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.325166 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.409541 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.410825 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.413229 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.422044 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.422894 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.438019 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.537635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.538985 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.539744 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.545094 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.548516 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.557358 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.582535 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.585170 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.589433 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:34.592257 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:34.592785 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.593639 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.601343 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.606771 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:34.607740 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:34.614902 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:34.678124 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:34.851048 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.852543 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.853392 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.859241 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.866603 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.873759 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.876716 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:01:34.888295 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.952085 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:34.957386 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:34.972205 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.973286 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:34.975075 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.984347 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:34.995141 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:34.995496 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:34.996325 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:34.996836 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.001848 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.003150 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.027727 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:35.052318 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:01:35.069687 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:35.114990 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.115355 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.117489 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.120791 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.124025 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.129067 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:35.129752 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:35.210763 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.225967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:35.226472 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.227864 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.232542 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.233799 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.234728 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.238689 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.334239 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.338356 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.343289 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:35.354151 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.354586 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.355782 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.357232 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.411924 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:35.413151 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.414247 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.421491 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.424484 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.424726 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.425911 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.516723 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.519011 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.526568 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.527920 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.529259 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.532887 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:35.559378 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:35.563295 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:35.630517 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.640614 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.644071 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.644774 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.645044 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.647351 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.653183 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.707226 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.708173 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:35.710203 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:35.718988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.721148 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.722980 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.735522 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:35.749094 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:35.780299 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.782853 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:35.788007 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.790135 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.790181 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:35.797713 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:35.809334 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.894381 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:35.895633 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:35.899003 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:35.899443 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.899993 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:35.900463 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:35.930043 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:36.009336 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.012068 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:36.016669 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:36.018275 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.019742 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.022898 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.041554 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.063570 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:36.083683 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.085635 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.088629 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.090186 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.097057 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.130087 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.158812 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:36.167910 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:36.175747 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.177592 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.181089 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.185106 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.188838 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.189513 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.245865 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:36.254101 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 20:01:36.281213 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.281533 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.283368 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.291301 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.300562 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.311175 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.365582 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:36.415400 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.422677 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.422809 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.434310 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.437049 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.440127 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.551175 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:36.564179 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:36.589459 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.589690 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.595925 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.602180 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.603399 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.604106 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.622781 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:36.638216 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:36.685010 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:36.710857 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.712946 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.714803 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.717233 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.721458 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.725121 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.768460 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.770742 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.777387 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.778744 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.787880 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.797375 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.810900 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:01:36.811975 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:36.858742 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.859103 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.862874 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:36.863004 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:36.869032 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.870085 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.879352 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.925136 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:36.958227 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:36.966421 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:36.967072 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:36.967164 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:36.969980 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:36.978127 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:36.998824 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:37.040395 16949 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:01:37.040524 16949 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:01:37.040948 16949 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:01:37.041972 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:37.042084 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:37.042131 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:01:37.042310 16949 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:37.072834 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:01:37.076803 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.080590 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.086498 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.086835 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.088307 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.100720 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:37.117739 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:37.132421 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.136708 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.139482 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.142575 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.144183 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.165601 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:37.186626 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:37.219076 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:37.244285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.244532 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.245694 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.246559 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.249197 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.265803 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.315538 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:37.330353 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:01:37.350107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.352219 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.356104 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.357777 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:37.369288 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:37.370075 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.370626 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.472827 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.475112 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.485894 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.486315 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.486941 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.490952 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.501571 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.504321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.509048 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.513142 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.515784 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.538983 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.595016 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:01:37.602672 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:37.624367 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.630568 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.630882 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.637863 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.645510 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.661095 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.692975 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:37.714978 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:37.742209 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:37.742538 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.743916 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.744998 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.749078 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.749090 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.778820 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.790745 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:37.817562 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:37.843652 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.844162 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.848645 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.852728 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.855295 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.857964 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:37.861791 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.911371 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:37.917057 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:37.917443 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:37.918911 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:37.924885 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:37.937511 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:37.942453 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:38.014464 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.021081 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.023399 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.024078 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.031077 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.042590 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:38.076706 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:38.140012 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:38.142365 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:38.150933 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.152875 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.153251 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.154253 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.159577 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.161957 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:38.171509 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:38.236558 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 20:01:38.245095 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:38.321316 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.325603 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.327435 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.328181 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.336004 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.350982 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:38.361701 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:38.425998 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.432217 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.435732 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.437616 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.443051 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.448073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:38.459084 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:38.459204 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:38.519135 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.520421 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.521944 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.523052 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.525512 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.528667 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.589514 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.589721 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.599543 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.601071 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.602635 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:38.617549 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:01:38.620432 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:38.684430 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:38.686398 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:38.706274 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.708859 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.709836 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.712179 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.713079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.718570 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:38.725220 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:38.754334 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:38.809072 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.812288 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.815362 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.818382 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.821529 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:38.821687 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:38.828639 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.879632 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.881740 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.883764 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.890105 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:38.890636 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.915300 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.968228 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:38.970438 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:38.970808 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:38.971065 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:38.977974 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:38.981133 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:38.994637 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:39.049593 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:01:39.083817 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.085994 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.086534 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.088055 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.108742 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.109108 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:39.151134 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:39.217105 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.217254 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.221673 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.225049 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:39.231009 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:39.234086 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:39.235793 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.245546 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.253849 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.256044 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.263535 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.268227 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.271216 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.290774 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:39.293212 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:39.367159 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.371310 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.371985 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.372069 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.373310 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.373680 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:39.391299 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:39.397533 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:39.474932 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.477733 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.479908 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.480083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.481242 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.489235 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.587328 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.589035 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.590646 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.594213 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.601840 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.609907 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.609910 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.610486 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.637703 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.643496 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.656785 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.660048 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:39.691612 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:01:39.693686 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:39.716306 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:39.732841 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:39.744737 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.746281 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.750767 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.753679 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.758608 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.759299 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:39.790015 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:39.816813 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:39.942620 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:39.946573 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:39.947656 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:39.949661 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:39.957268 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:39.967546 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:39.994884 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.044831 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:40.053577 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:40.056751 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.057523 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.057871 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.059298 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.060039 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.061031 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.102627 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:40.137420 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:40.160773 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.162830 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.163671 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.164076 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.165756 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.168090 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.182391 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.273648 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.275060 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.280860 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.282346 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.289196 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.308836 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:40.327454 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.332424 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.338023 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.338407 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:40.340691 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:40.343401 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.345124 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.353751 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.407038 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:40.451454 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.452332 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.452862 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.458425 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.458774 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.472988 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:40.539359 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.552407 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.553670 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.561913 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.563041 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.574065 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.575217 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.606986 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:40.617568 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:40.630519 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:40.635727 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.641201 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.641326 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.656016 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.658149 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.683996 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.700717 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:40.721280 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.724560 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.731912 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.732996 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.740947 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.754886 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.785228 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:40.832432 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.836186 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:40.837080 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.839241 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.853262 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.861099 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:40.891981 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:40.942885 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:40.953101 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:40.953383 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:40.960798 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:40.966336 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:40.966828 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:40.968169 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.008637 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.013244 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.016662 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.020056 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.022330 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.041630 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:41.041666 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:41.107887 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.108582 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.109117 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.113070 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.118757 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.125908 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:41.128214 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:41.210629 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.211002 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.213862 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.214757 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.224053 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.230628 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:41.230823 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:41.237442 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:41.307850 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 76 similar messages]
I20240501 20:01:41.323156 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.324245 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.332150 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.336115 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.341356 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.344004 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.384590 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.385627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.386200 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.387945 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.414752 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.422039 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:41.424191 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:41.425201 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:41.478776 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.480471 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.482656 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.487442 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.500147 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.500411 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:41.553577 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:41.607864 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:41.645074 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:01:41.657272 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:41.705765 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.707886 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.708170 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.710400 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:41.714882 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:41.728893 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.733147 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.800215 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.802139 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:41.804075 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.808377 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.813040 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.814306 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:41.845738 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:41.877225 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:01:41.901597 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:41.903599 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:41.904552 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.912954 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:41.915968 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:41.916100 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:41.918982 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:41.956498 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:42.033950 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.034469 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.038872 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.042176 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.046043 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.046370 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.048187 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:42.058925 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:01:42.061446 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.065876 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.066782 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.093608 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.110855 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:42.136461 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:42.151971 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:42.186699 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.187321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.189796 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.195549 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.204144 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.213883 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:42.223058 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:42.272321 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:42.289875 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.290124 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.295300 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.296026 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.299247 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.318665 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:42.327219 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:42.415117 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.426934 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.428676 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.436120 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.441740 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.444324 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.448370 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.457702 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.464062 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.476526 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.477857 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.484704 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:42.524041 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:42.527091 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:42.529784 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:42.532533 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:42.559432 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.563411 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.567234 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.567726 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.573024 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.577653 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.668179 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.671348 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.671835 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.673606 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.674990 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.690373 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:42.700466 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 20:01:42.701125 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:42.785395 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.788849 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.790979 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.802448 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.812387 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.814863 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:42.821072 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.821691 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.822628 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:42.822849 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:42.824817 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:42.829598 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.841041 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.847975 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.882079 16949 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:01:42.882179 16949 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:01:42.882558 16949 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:01:42.883618 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:42.883811 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:42.883883 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:01:42.884018 16949 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:42.898077 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:42.940148 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:42.941588 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.943181 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:42.945152 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:42.945684 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:42.947788 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:42.965488 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.021462 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:43.022094 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:43.027202 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:43.041217 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.043139 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.043941 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.044700 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.061077 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.075263 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:43.114457 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.127043 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:43.141479 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:43.154291 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.156512 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.161715 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.165963 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.168166 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.169330 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.213479 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.225030 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.225577 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.226265 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.226537 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.228296 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:43.370260 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.377630 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.405853 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:01:43.485594 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.490942 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.493770 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.501189 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.512102 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.521451 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:43.553517 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:43.564949 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.566936 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.568256 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.569932 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.577087 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.577548 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:43.594547 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:43.665457 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:43.665709 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:43.670847 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.674468 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.674583 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.676087 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.682803 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:43.697746 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.700641 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:01:43.725898 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:43.773286 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.780261 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.785913 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.789076 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.796629 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.798530 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.862632 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.862746 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.865844 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.866636 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:43.866909 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.880343 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.933692 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:43.934847 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:43.936867 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:43.942211 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.945086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:43.945327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:43.954550 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:43.980099 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:01:43.990006 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:44.025314 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:44.035054 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.035265 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.035338 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.036170 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.036965 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.061673 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:44.063874 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:44.153414 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.157440 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.159092 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.162277 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.167821 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.175172 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:44.218787 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:44.244442 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.244477 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.252925 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.253896 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.255219 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.260181 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.318189 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.321658 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.322736 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.323076 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.323508 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:44.329521 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:44.329538 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:44.370018 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:44.422396 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:44.425617 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:01:44.427363 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.429250 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.440187 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.442124 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:44.445227 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:44.446419 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:44.473613 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:44.541122 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.551468 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.552824 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.553280 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.553656 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.566022 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.608142 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.612107 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.612766 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.627290 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.627367 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.627583 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:44.634059 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:44.638324 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:44.713917 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.716234 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.717516 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.718465 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.724824 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.727124 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:44.808630 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:44.814407 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.815182 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.819859 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.828173 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.830121 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.845909 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.945839 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:44.946535 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.947945 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.958985 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.961786 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:44.963179 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:44.971287 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:44.972570 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:44.985805 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.001194 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.003499 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.005235 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:45.085139 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:45.095863 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:45.254280 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:45.288286 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.315936 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.324239 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.325918 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.327499 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.331573 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.346693 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.349922 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.358247 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:45.362309 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.366338 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:45.372849 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.378108 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.378121 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.400486 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:45.432467 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:45.434306 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:45.481114 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:45.484025 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.487426 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.488754 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.489149 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.495833 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.496201 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:45.517398 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:45.530772 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:45.551398 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:45.565106 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.581763 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:45.585472 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:45.587421 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.589704 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.593076 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:45.597239 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.600713 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:45.607565 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.705485 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.721371 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.723821 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.724694 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.725003 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.725137 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.729866 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.736378 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:45.759361 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.762454 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.762543 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:45.775975 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:45.776894 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:45.818418 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:45.856346 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.857077 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:45.859076 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.863162 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.863320 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.863482 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:45.881536 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:45.964143 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:45.965068 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:45.966390 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:45.967283 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:45.967283 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:45.983369 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:46.055622 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:46.071110 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.071790 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.075033 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.085019 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.099104 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:46.108851 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.137006 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.143816 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.146090 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.155577 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.155597 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.158080 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.176754 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:46.184711 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:01:46.248349 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:46.256796 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.257005 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.261562 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.265059 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.271976 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.272559 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.278128 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:46.355993 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 20:01:46.356704 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:46.361754 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.367202 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.370376 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:46.373081 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.378146 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.378608 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.453369 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.459805 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.460187 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.467722 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.469063 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.481766 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.489771 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:46.527228 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.529461 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:46.530303 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.531016 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.540170 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.548135 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:46.579525 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:46.579844 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:46.623709 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.627286 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.634042 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.641616 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.644320 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.647241 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.657006 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:46.684444 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:01:46.737618 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.742897 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.752218 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.752882 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.757179 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.768115 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.779883 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:46.816234 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.816313 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:46.817124 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:46.818850 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:46.819666 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:46.819694 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:46.868710 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:46.882040 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:46.898610 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:01:46.906354 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:47.001480 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:47.063108 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.069824 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.071785 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.073266 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.082865 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.083415 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.180883 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.186678 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.201470 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.203259 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.208168 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.225064 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:47.225383 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:47.238976 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:01:47.244140 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.245404 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.247321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.249781 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.255569 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.257440 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:47.289508 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:47.374034 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.377525 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.378399 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.379488 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.381075 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.381934 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:47.482271 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:47.482772 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.484303 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:47.487725 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:47.487923 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.488158 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.504961 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.509488 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.569727 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.574545 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.582348 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.583710 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.583714 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.591584 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.647042 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.647847 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.656424 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.657305 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.660722 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.661697 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:47.678115 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:47.701720 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:47.732452 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:47.742830 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:47.748122 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:47.751487 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.756610 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.757395 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.757539 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.761273 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.765213 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:47.793934 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:01:47.809319 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:47.863996 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.869983 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.870576 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.881537 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.882207 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.883023 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:47.935011 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.942242 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:47.943145 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:47.949060 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:47.950148 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:47.956027 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.016875 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.021456 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.022197 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.022526 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.022606 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.029242 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:48.042304 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:48.054214 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:48.071012 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:48.086616 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:48.127410 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.130900 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.131850 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.133705 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.135669 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.147465 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:48.150279 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:48.168968 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:48.235636 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:48.257123 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.258496 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.262434 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.264321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.270393 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.270654 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.284482 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:48.287315 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:48.301378 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.305696 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.307921 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.309718 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.321519 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:48.333873 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:48.394603 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.395668 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.396219 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.401638 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.408644 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.409117 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:48.446586 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:01:48.482174 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:48.504053 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.504835 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.508056 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:48.508272 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:48.509019 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.513834 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.525591 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:48.585059 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:48.611582 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.615077 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.616955 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.621034 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.621287 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.621958 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:48.657259 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:48.680457 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:48.685657 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:01:48.779211 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:01:48.809980 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.810712 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.811344 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.815250 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.824580 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.832041 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.916630 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:48.918747 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.925269 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.925514 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.929293 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.948326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:48.981550 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:48.987041 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:48.992265 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:48.993037 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:48.994833 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:48.996742 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.012548 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.082929 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.087281 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.090345 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.090543 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.093613 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:49.099198 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:49.110849 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:49.153012 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:49.165753 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:49.191448 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.192162 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.200639 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.204468 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.206732 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:49.215998 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:49.219183 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:49.301844 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:49.326666 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.326953 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.334067 11355 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.334280 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.334410 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:49.335565 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:49.335870 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.337924 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.341636 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:49.349520 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:01:49.367899 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.370575 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.370851 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.396672 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.471334 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.471630 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.473737 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.475045 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.477257 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.477283 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:49.479413 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:49.504853 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:49.577729 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:49.578191 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.579772 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.583734 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.595099 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.597824 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.597841 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:49.687213 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:49.700500 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.701786 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.701889 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.716596 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.717761 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.732430 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.748409 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.757644 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.758141 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.763590 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.767304 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:49.769220 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:49.775970 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.863942 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.864277 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.866850 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.867118 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:49.874104 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.890511 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.971940 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:49.974064 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:49.974490 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.974751 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:49.982878 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:49.998667 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.056723 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:50.059352 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.060607 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.062682 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.073642 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.081606 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.085793 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.090975 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.137396 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.143117 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.144107 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.144873 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.146045 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.155948 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.169962 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:50.212502 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:50.224442 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.247135 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:50.247542 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.248049 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.249082 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.250530 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.258132 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.262727 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.289244 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:50.314110 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.375391 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.375391 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.376514 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.379372 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.379822 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.392624 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.550133 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.553669 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.554211 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.562327 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.563058 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:50.564786 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:50.565867 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.571563 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:50.609081 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:50.659552 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.660260 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.661593 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.670738 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.671268 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.679271 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.737430 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:50.742012 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.742854 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.754884 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.761045 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.762965 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:50.767354 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:50.778291 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:50.804666 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.821605 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.824831 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.827435 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.834134 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.835158 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.837481 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:50.894307 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:50.919951 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:50.926963 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:50.930250 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:50.932866 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:50.933075 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.935189 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:50.941210 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:50.974109 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:50.984979 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:51.025259 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:51.034648 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.040625 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.044014 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.045114 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.045156 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.058310 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.121537 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.124954 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.126663 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.131098 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:51.134829 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.141867 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:51.153098 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:51.155162 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:51.184779 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:51.211644 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.212298 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.217931 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.218416 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.219321 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.219772 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.304209 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:51.315296 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.319144 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.320317 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.323053 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.323169 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.345795 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.448168 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.454279 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.455565 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.458809 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.469786 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.470623 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.484517 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.489106 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:51.491269 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 20:01:51.492926 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.513698 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.517508 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.530081 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:51.558792 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:51.595896 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.596995 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.597832 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.598264 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.608868 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:01:51.610042 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.626603 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.705480 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:51.707554 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.708477 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.713886 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.717114 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.720777 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.726768 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.747814 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:51.824304 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:51.828406 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:51.835568 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.841109 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.841445 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.845458 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.862437 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.865602 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.869092 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.879189 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.881500 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.905272 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.905864 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:51.912642 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:51.985248 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:51.987272 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:51.988047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:51.988413 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:51.989568 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:51.993119 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:52.011775 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:01:52.057591 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:52.097273 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:52.151085 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:52.173864 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.174140 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.176301 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.177594 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.178292 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:52.193874 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:01:52.194886 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:52.202167 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:52.237421 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:52.287508 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.288192 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.289532 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.292831 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.301652 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:52.308866 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:52.308959 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:52.317438 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:52.335242 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:01:52.390616 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.393751 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.405259 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.406441 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.406543 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.406927 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:52.442958 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:52.506407 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:52.515477 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.518544 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.527683 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.529886 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.531391 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:52.540084 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.566773 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.575616 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:52.577829 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.579447 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.586376 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:52.605785 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:52.610646 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:52.611617 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:52.623595 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:52.676059 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:52.692718 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.694592 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.697000 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.697225 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:52.700286 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.701997 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.791452 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.793413 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.797461 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.805449 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.806229 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.808099 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:52.848819 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:01:52.902021 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.906075 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.909315 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:52.911443 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:52.913916 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.929934 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.930198 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:52.954645 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:52.955515 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:52.961755 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:52.971243 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:52.971357 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:52.980024 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:52.987051 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.066687 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.074396 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.079936 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.080519 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.090993 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:53.103143 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:53.107596 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:53.158058 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:53.193748 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.193936 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.196496 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.201056 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.201969 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:53.202898 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:53.212453 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:53.266974 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:53.283507 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:53.283527 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:53.301424 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.304164 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.305994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.306037 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.306281 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.345117 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.355150 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:53.358455 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:53.359659 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.360664 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.366992 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.369263 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:53.373780 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:53.384368 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:53.387641 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:53.470909 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.472257 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.475962 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.484720 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.489732 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.495692 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:53.517192 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:53.527261 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240501 20:01:53.571164 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.572988 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.573144 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.574743 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.583225 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.589751 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:53.649821 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:53.678761 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.683816 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.693413 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.695690 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:53.696173 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.707263 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.730901 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:53.738206 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:53.738987 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:53.752460 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:53.752802 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:53.754436 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:53.755139 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:53.801017 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:53.814400 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:01:53.991659 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:01:54.022101 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:54.058033 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.058298 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.061931 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.062052 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.064299 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.078979 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.103546 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.107295 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:54.109097 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:54.113626 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.119820 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.120366 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.125638 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:54.142536 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:54.160606 23051 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:01:54.160704 23051 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:01:54.161060 23051 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:01:54.161810 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:54.165904 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:54.165966 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:01:54.166118 23051 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:54.197816 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:54.216224 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.216432 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.219274 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.220432 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.221920 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.227452 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:54.259397 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:54.319321 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.319684 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.324901 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.325497 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.333101 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.334831 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:54.398630 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:54.443078 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.444362 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.449410 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.451833 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.461385 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.464151 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.465175 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.469070 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.472491 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.479584 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.485944 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.501780 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:54.559254 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:01:54.598156 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.600533 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.600633 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.601058 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.602910 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.611981 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:54.629034 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:54.709630 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.709921 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.712782 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.715070 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.720614 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.723304 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:54.737861 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:54.758352 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:54.772197 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:54.785851 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:54.792721 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:54.814218 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:54.824039 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.825441 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.832449 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.846246 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.852473 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.858354 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.860086 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:54.860347 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:54.880015 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:54.883575 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.892207 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.894814 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.894869 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:54.905841 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:54.984486 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:54.984817 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:54.987254 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:54.993136 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:54.997244 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.004269 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.076589 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:55.095108 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.095207 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.095194 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.101065 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.114972 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.134557 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:55.134781 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.183554 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.188549 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.194113 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.195533 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.197386 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:55.205471 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:55.228771 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.239229 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:55.261138 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.268036 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.268949 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.274183 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.280243 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.286419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.363116 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.364112 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.365528 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.366998 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.369112 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.381398 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.416205 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:55.484447 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.490695 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.490764 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.492235 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.496826 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.508112 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.566419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.566897 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.569949 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.572590 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.572819 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.593281 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.644933 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:55.667922 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:55.678634 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:55.681809 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:55.709270 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:55.795624 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:55.796568 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.799963 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.800042 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.801667 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.802362 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.811165 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:55.826171 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:55.846364 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:55.871282 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:55.910010 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.910367 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.929913 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.934065 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.938597 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:55.938979 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.948843 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.950212 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.969374 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.970124 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:55.977861 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.978197 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:56.004438 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:56.072697 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.075191 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.075577 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.083020 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.084605 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:56.104238 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:56.104769 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:56.110242 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.187798 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.194500 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.197312 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.200429 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.204015 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.214378 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:56.223429 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:56.303567 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:56.312040 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:56.316110 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.319288 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.319928 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.332342 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.343485 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.351251 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.351301 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.353091 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.355320 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.361614 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.371330 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:56.382982 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:56.465893 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.467617 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.470952 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.473850 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.479616 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:56.491853 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240501 20:01:56.493551 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.574028 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.575384 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.576896 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.579852 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.579959 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.591104 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.675908 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.678171 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.678985 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.680121 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.682852 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:56.697634 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:56.703723 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.741796 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.743782 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.748941 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:56.750198 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:56.750314 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:56.751111 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.752661 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.760986 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:56.813556 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:01:56.852084 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.854812 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.854961 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.856472 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.862138 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:56.865060 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:56.873888 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:56.924863 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:01:56.933624 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:56.970074 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.970957 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.971555 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:56.978821 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:56.982789 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.003423 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.024524 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:57.031733 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:57.039357 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.039690 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.042893 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.043164 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.075974 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.085258 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.129581 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.130936 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.132455 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.133195 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.134182 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.139334 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:57.162274 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:57.228788 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.231261 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.231725 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.238809 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.239467 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.246536 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.247651 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:57.256232 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:57.299657 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:57.350667 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:01:57.370424 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.371284 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.372707 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.376431 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.379050 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.397925 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.403867 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:01:57.420395 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:01:57.468892 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:57.509801 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:57.509860 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:57.535007 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.537432 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.540154 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.540778 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.541098 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.561394 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.642980 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:57.649780 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:57.650492 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.657048 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.657938 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.672256 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.675148 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:57.703272 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:57.719115 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:57.741258 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.741801 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.744405 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.750296 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.758927 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.767124 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.785727 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:57.810750 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.813438 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.815675 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.815968 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.816669 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.830082 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:57.830624 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.922008 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:57.922338 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:57.924212 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:57.925850 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:57.934020 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:57.934607 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:57.975324 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:01:58.021620 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.030063 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.030699 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.032434 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.034044 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:58.042598 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:58.069413 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:58.073122 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:58.110993 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:58.112514 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.114403 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.115592 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.120594 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.125705 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.135763 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.200403 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.200721 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:01:58.210847 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:58.213696 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.213989 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.214334 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.215134 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:58.221925 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:58.304041 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.309235 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.311538 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.312676 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.313820 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:58.322547 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:58.326990 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:58.337932 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:58.429108 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.436676 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.444697 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.448837 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.449955 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.458107 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.468542 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.477782 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.488272 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.499100 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:58.499902 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:58.506045 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.509054 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:58.546367 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:01:58.553018 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 20:01:58.586970 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.588881 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.589277 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.591336 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.599933 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.602294 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:58.684530 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:58.697469 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.705641 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.706512 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:58.707156 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:58.709482 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.711766 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.717883 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.812930 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.813020 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:58.821691 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:58.823112 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:58.824009 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:58.828399 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:58.832366 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.833043 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.833102 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.837466 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.838390 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.845634 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.861176 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.884186 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.887176 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.959117 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.961897 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:58.965245 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.965514 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.973188 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:58.977213 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.117941 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:01:59.173758 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.176852 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.177660 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.189858 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.194454 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:01:59.196878 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.209062 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:01:59.222816 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:59.282374 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.288516 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.293287 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:59.293769 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.294617 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.296312 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.297616 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.320251 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:59.336422 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:59.374579 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:01:59.382807 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.389240 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.390655 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.398005 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.398891 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.402972 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.511617 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.516448 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.519305 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.520515 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.535398 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.538779 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.539052 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.541908 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:59.566211 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:59.567384 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.570384 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.573256 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.587050 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.664793 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:59.665479 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.668421 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.669400 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:01:59.673456 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:59.675532 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.689988 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.690755 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.731395 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:01:59.743896 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:01:59.758833 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:01:59.770917 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.772349 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.772599 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.773814 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.777990 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.785034 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.787454 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:59.807739 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:01:59.864969 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.867717 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:59.875485 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:01:59.875756 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.876722 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.882023 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.902554 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.924506 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:59.929879 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:59.933815 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:01:59.935963 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:59.936214 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:59.940600 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:01:59.962939 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:59.989310 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:02:00.024327 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.026456 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.031975 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.033049 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.033898 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.036656 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:00.063212 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:00.108518 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:00.139966 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:02:00.140705 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.141503 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:00.143528 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:00.143910 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.146035 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.148427 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.154438 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.249039 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.249709 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.250468 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.257107 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.281491 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.291082 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.297346 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.303267 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.305837 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.312317 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.313562 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.315429 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:00.325613 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:00.415707 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.416846 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.421859 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.421911 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.429219 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.436070 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.518695 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.519620 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.524554 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.531364 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.534420 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.549388 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.625738 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.628679 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.640410 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.648509 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.650736 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.661937 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.689576 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:00.692042 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:00.693560 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:00.694531 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:00.704298 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:00.705356 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.707882 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:00.709944 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:00.727320 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:00.833271 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:00.891883 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:00.941516 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:00.975767 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:00.992743 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:00.994159 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:00.996582 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.003461 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.004973 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.007609 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.054484 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.054570 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.056983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:01.058885 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:01.068677 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.071327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.073186 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:01.095378 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:01.097342 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:01.163056 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.166510 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.168226 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.169716 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.173238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.176751 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:01.238978 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:01.254156 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:01.274262 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.284768 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.286557 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.298086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.305101 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.318428 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:01.324100 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:02:01.361918 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.363699 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.364114 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.365057 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.371266 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.382318 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.448021 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.449478 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.450014 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.452004 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:01.456262 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:01.459165 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.481396 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:01.511880 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240501 20:02:01.548553 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.554286 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.555948 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.556175 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.558646 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.561265 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:01.585158 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:01.646597 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:01.654438 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:01.667449 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.670909 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.675704 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.676856 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.677387 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.689383 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.731070 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.734292 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.736568 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.738425 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.741001 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.766417 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.820614 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:01.825038 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.825382 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.829751 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.834863 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.839888 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:01.841459 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:01.931000 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:01.931092 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:01.931777 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:01.932961 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:01.935303 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:01.953454 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:02:01.954108 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:01.997756 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.039072 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.040868 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:02.040997 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:02.042229 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.043218 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.044766 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.045578 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:02.081777 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:02.106568 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.111255 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.112110 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.115581 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.135382 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:02.142797 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:02.145434 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:02.159233 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:02.190742 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:02.195102 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.203826 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.208124 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.209697 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.211066 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.213490 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.219002 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.302500 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.302758 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.304553 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.304555 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.306417 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:02.313777 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.319550 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:02.325706 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:02:02.337009 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:02.384703 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.421082 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.421833 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.426944 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.428248 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:02.441041 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:02:02.448843 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.449113 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.482620 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.483280 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.486312 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.486775 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:02.496284 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:02.497000 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:02.503283 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.505255 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:02.555011 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:02.558687 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:02.677902 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.737731 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.739399 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.741734 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.747992 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.758505 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.759469 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.826287 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.826929 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:02.831045 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.839804 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.844993 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.849170 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.900228 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:02.901154 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:02.902014 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:02.903424 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:02.905978 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:02.906515 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.919956 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:02.999562 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.000097 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.007205 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.011126 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.013689 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.023218 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.108875 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.113436 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.120077 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:03.122049 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 20:02:03.122874 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:03.130746 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.137959 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.139616 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:03.174166 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:03.200189 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.202039 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.205332 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:03.205886 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:03.211396 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.213057 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.235750 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:03.243522 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:03.245927 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:03.270922 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.273046 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.275871 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.281641 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.283044 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.289290 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:03.321892 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:03.371273 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.375304 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.375677 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.376253 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.393834 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.393879 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:03.462756 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:03.482241 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.491199 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.492972 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.493393 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.493734 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.494469 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:03.564174 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:03.574249 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.574903 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.577440 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.584277 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.587648 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.604662 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.637270 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:03.642921 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:03.642943 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:03.650828 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.650851 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.654407 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.661418 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.671613 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:03.695291 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:03.755847 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.755961 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.756446 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.762916 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.763330 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:03.763587 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:03.766057 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:03.786211 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:03.812340 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:02:03.812855 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:03.859932 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.862561 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.871981 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.872733 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.882694 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:03.883198 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:03.927721 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:03.951364 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:03.952528 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.953400 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:03.953770 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:03.957960 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:03.970891 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.014719 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.019369 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.027576 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.027822 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.029090 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.030974 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.124162 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.124454 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.128513 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.131503 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.141162 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.146080 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:04.164577 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:04.201674 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:04.205955 27907 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:04.206063 27907 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:04.206460 27907 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:04.207305 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:04.208091 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:04.208176 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:04.208477 27907 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:02:04.220968 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:04.266690 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.266923 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.267802 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.269963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.272977 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.273849 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:04.379590 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:02:04.408313 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:04.416548 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:04.429217 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.430418 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.436786 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.443840 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.444420 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.447640 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.529359 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.531203 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.533455 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.533797 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.549054 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.555752 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:04.561942 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:04.610653 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:04.614085 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:04.616789 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.621788 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.624123 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.639598 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.645072 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:04.645381 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:04.657832 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:04.678548 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:02:04.701011 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.701387 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.703776 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.715139 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.715570 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.716519 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.807276 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.808892 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.809131 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.809386 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:04.810943 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:04.818442 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.818531 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:04.833856 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:04.916283 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.921761 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.923344 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:04.926661 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:04.928092 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:04.939536 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:04.941869 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:02:04.984800 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:04.993731 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:04.998411 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:04.998718 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.001304 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.022284 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.025849 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:05.037837 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:02:05.048558 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:05.077870 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.081583 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.088588 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.090687 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.104508 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:05.106833 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:05.111449 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:05.186795 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:05.190243 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.190476 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.194005 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.205372 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.209935 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.218318 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.305442 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.306797 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.314194 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.315526 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.321512 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.321717 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:05.347565 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:05.367826 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.373121 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.373142 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.378885 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.382287 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.383201 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.462185 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:05.467825 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:05.468385 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.469127 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.470121 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.471575 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.474980 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:05.543190 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:05.544275 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:05.558068 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:05.579469 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.585119 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.585386 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.585743 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.588932 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:05.597836 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:05.598194 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.703320 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.707758 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.708014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.717643 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:05.720793 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:05.722152 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:05.727455 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.730731 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.732463 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.735829 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.737237 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.744802 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.771135 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:05.839875 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:02:05.847561 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:05.852362 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:05.855268 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.857131 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:05.858819 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:05.865693 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:05.873427 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.055229 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.055502 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.058260 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.065680 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.068022 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:06.070616 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.097947 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.098227 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:06.158939 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.162755 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.163606 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:06.166515 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:06.166710 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.171813 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.194018 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:06.201300 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:06.230741 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:06.264070 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.266793 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.267642 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.273947 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.287573 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.289929 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.377070 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.381443 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.382704 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.383787 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.386040 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.393932 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.427130 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:06.438714 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.442629 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.449108 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.453979 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.458300 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.459345 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:06.486793 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:06.535380 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.537024 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.540452 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.542655 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.548247 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.550490 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.589373 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:06.591428 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:02:06.632566 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:06.637173 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.642976 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.644779 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.645215 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.661263 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.662088 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.719743 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:02:06.760005 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.761766 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.767933 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.772531 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.777029 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.796502 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.802640 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.804558 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.813129 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:06.814061 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.816419 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.820282 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.824215 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:06.837457 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:02:06.896567 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:06.917711 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:06.918758 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:06.922781 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:06.923915 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:06.925853 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:06.942596 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:06.954041 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.001119 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 20:02:07.019286 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.020895 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.024451 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.029057 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.031337 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.033980 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:07.115890 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.129114 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:07.134479 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.139623 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.142166 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.165602 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.167140 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.170178 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.174839 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.176213 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.180533 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.185504 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.202787 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.204600 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:07.221051 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:07.293269 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.294957 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.298494 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.301103 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.309247 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.309597 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:07.315141 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.318411 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.342471 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:07.360843 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:07.394363 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.398368 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.399046 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.405107 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.407471 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.412338 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:07.503211 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:02:07.504498 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:07.510350 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.513456 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.518062 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.520750 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.521073 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.524837 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:07.540112 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:07.551192 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:07.554095 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.558885 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.563232 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.572821 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.574093 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.581537 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:07.660627 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.674721 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:07.694326 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:07.730949 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:07.851341 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.856153 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:07.859545 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.861037 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.865944 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.886739 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.914836 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:07.920297 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:07.924517 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.925244 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:07.928436 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:07.938577 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:08.016247 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.016604 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.016985 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.018898 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.019316 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.028406 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.044454 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:08.054054 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:08.082536 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:08.130569 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.135972 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.137054 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.138617 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.140851 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.149055 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:08.171451 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:02:08.172111 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.214756 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.220630 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.221549 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.226119 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.233562 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.255751 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.294677 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.295274 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.295660 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.301419 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.301798 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.302670 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:08.313694 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:08.342345 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:08.343065 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.386921 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.390800 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.392453 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.395191 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.413254 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.415499 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:08.430907 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:08.487766 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.499100 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.503551 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.505555 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.509640 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.510993 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.520292 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:08.542968 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:08.553294 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.586367 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.588114 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.592020 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.592680 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.596050 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.619508 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:08.661142 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:08.664613 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.667755 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.671653 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.671877 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.673408 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.675788 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.768175 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.768383 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.772647 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.773313 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.783712 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.785363 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.873154 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:08.874886 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.880319 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.881084 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.889096 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:08.889866 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:08.894133 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:08.916407 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:08.950006 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:08.951161 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:08.966687 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.973342 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:08.981011 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:08.981729 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:09.015733 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:09.029058 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.030151 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.033674 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.035229 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.035737 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.046907 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:09.105329 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 20:02:09.131678 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.132360 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.137028 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.137449 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.140054 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.145656 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:09.164117 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:09.218824 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:09.219568 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:02:09.237994 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.245077 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.250896 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.251329 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.254729 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.269985 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:09.297420 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:09.313709 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:09.327437 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.330722 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.331892 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.334167 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.344507 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.350250 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:09.352718 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:09.366782 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:09.380110 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:02:09.511857 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:09.543006 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:09.565176 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:09.565676 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.566124 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.566704 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.567085 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.568006 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.568995 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:09.644563 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:09.644661 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:09.650732 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:09.687461 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:09.688581 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.689783 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.691978 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.695897 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.703912 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.715358 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.716771 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.721068 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.723300 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.724889 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.726079 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.738793 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.835494 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.838272 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.846592 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.847245 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.848058 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:09.861505 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:09.914803 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:09.939339 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:09.941154 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.945904 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:09.950604 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:09.957185 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:09.957616 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:10.017890 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.051964 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.058462 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.065661 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.073153 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.073377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.088521 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.102115 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.103915 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.112293 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.127079 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.129573 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:10.132267 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:10.140229 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:10.183372 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.218056 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.221516 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.224188 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.225994 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:10.226222 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.241202 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.242087 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:10.298882 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:10.323427 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.324113 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.328215 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.328630 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.329326 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.329520 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:10.358523 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:10.373394 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:10.394034 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.434186 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.436028 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.436411 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:10.439836 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:10.454541 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.456677 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.467942 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.494722 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.497143 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.497692 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.498726 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.500037 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.507527 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.594530 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.595836 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.601938 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.605535 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.612686 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.621140 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:10.642660 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:10.644623 29754 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:10.644709 29754 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:10.645100 29754 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:10.645988 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:10.647300 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:10.649149 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:10.649230 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:10.649433 29754 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:02:10.707901 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.709012 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.718147 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.719718 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.720666 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.726519 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.799957 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.806022 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.806728 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.807189 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.808202 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.809609 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:10.857507 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:10.871065 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.873726 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:10.876473 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.876760 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.880565 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:10.891011 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:10.898756 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.900498 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:10.914742 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:10.921043 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:10.985627 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:10.988227 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:10.989121 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:10.992252 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:10.996104 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.006475 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:11.027609 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:11.030648 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:11.120630 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.122318 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.123001 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.125118 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.125471 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.130429 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:11.227257 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:11.297808 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.301010 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.302979 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.309870 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.313488 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.322130 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:11.372241 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:11.407544 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.412724 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.413486 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.418306 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.420156 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:11.428047 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:11.441159 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.475765 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.476018 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.479650 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.479673 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.482668 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:11.487648 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:11.495637 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.571722 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.573725 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.576478 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.579986 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.580554 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.588171 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:11.615569 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:02:11.633805 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:11.679309 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.685199 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.686057 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.687664 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.693991 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.697052 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:11.719309 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:11.797797 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.800102 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.802699 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.804383 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.805202 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.808050 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.815673 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.816774 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:11.818110 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:11.819764 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240501 20:02:11.821667 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.822319 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.826556 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.853821 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:11.866331 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:11.892858 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:11.945468 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:11.948593 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:11.949357 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:11.950239 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:11.956787 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:11.961514 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.048370 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.048468 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.051224 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.051540 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.056402 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:12.059585 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:12.065945 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:12.117455 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:12.119596 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:12.138604 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:02:12.159763 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.175122 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.179029 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:12.182328 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.191190 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.192203 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:12.194617 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.197007 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.199594 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.205144 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.207537 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.222210 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.224592 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.254920 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:12.311324 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.322258 31195 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:12.322391 31195 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:12.322824 31195 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:12.325358 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:12.325443 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:12.325513 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:12.325735 31195 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:02:12.333761 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.335716 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.338709 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.340444 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.348310 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.349829 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.443259 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.444594 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.445552 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.446022 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.447582 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.449275 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:12.511029 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:12.511667 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:12.543424 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:12.549341 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:12.550439 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.551645 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.556192 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.570474 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:12.571316 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.585255 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.601799 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.603256 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.607510 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.611063 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.616127 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.619181 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:12.681779 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.722681 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:12.723531 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:12.725203 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.727156 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.729169 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.753340 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:12.764403 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:12.764973 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:12.803283 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:12.814852 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:12.818974 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.898031 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:12.898089 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:12.904701 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:12.909278 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.910169 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:12.912483 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:12.916327 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.030115 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.033638 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.033686 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.033890 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.047525 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.056113 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:13.072588 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:13.080143 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:13.139089 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.143239 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.144074 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.146226 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.149235 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.150836 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:13.185237 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:13.241499 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.245411 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.248462 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.252275 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.260370 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:13.273169 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:02:13.281847 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.290215 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.304543 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.305632 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:13.310611 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:13.315479 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.319245 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.335014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:13.386898 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:13.413446 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:13.419915 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.424891 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.428637 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.429584 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.430083 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:13.438527 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:13.445984 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.520001 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.522197 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.522930 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.524847 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.528301 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.534245 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.613219 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.615797 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.616850 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.621014 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.621773 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:13.654397 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:13.656474 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:13.659428 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:13.686856 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:13.692534 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.693274 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.696327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.698071 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.698787 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.719749 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.800714 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.803239 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.804019 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.807314 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.807973 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.813066 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:13.815049 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:13.853866 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:13.860633 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:13.909677 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.915975 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:13.918367 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.929869 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:13.938531 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:13.939522 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.991493 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:13.992691 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:13.997231 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:14.001310 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.006597 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.011014 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.064016 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.064788 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.068357 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.082499 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.088439 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.097599 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:14.126457 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:14.130792 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 20:02:14.131390 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:14.173200 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.177026 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.177210 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.182216 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:14.186956 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:14.188709 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.189090 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:14.228055 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:02:14.295892 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:14.296674 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:02:14.296756 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.298444 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.303067 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.310904 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.321079 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.361712 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.367223 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.370127 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.370913 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.372826 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.375293 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:14.397763 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:14.413136 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:14.455722 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:14.457576 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.457911 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:14.458771 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.459456 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.463006 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.474560 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:14.577062 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:14.622912 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:02:14.665450 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:14.670794 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:14.689100 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.690284 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.691696 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.693117 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:14.693220 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.704325 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:14.741976 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:14.762001 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:14.780730 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.784916 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:14.786820 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.789463 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.792019 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.792260 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:14.798115 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:14.879738 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:14.881214 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.881585 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.889348 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.889940 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:14.904747 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:14.985965 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:14.989545 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:14.997802 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.009166 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.010602 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.013828 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.047859 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.048432 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:15.051101 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:15.055439 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.058439 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.059870 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.062482 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:15.120836 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:15.126400 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:15.153637 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.159567 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.160745 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:15.162830 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:15.163187 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.169397 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.174904 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:15.214802 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:15.247280 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:15.258426 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.260529 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.263177 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.278158 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.280620 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:15.282682 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:15.283800 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:15.299978 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:15.359475 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:15.372582 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.378446 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.380627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.383780 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.386495 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.395072 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.411660 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.412566 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.414665 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.438102 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.443885 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.446350 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.534937 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.535938 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.541864 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.549247 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.549955 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.565608 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:15.565799 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:15.573637 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:15.670182 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.672806 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.681205 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.682775 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.684327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.684365 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:15.696038 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:15.755497 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.758931 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.759905 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.760491 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.770849 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:15.784711 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:15.794394 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.815860 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:15.817606 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.819614 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.827479 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.832762 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.848140 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.925725 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.927651 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:15.930716 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:15.931152 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:15.941401 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:15.944267 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:15.953964 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:15.980425 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:16.026660 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.026961 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.028393 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.029794 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.034184 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.036582 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:16.079813 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:16.089620 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:16.118490 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:16.146564 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.155187 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.158396 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.163308 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:02:16.167068 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:16.171098 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.171159 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.183178 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.187405 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.189805 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.192981 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.194756 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.213666 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.233024 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.288560 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:16.453997 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:16.485826 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:16.508718 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.514057 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.515359 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.516772 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.524715 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.537328 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:16.543515 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.572319 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.573483 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.574190 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.574550 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.581683 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.582710 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.673358 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.674182 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.675439 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.682209 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.683167 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:16.684453 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.685089 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.764317 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:16.774510 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.777402 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.781715 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.783994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.784304 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.796262 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.855592 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:16.865111 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:16.879935 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:16.882103 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.882812 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:16.900043 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:02:16.901543 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.904842 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.919471 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.925045 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:16.925889 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:16.930846 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:16.936517 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.936863 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:16.937448 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:16.984290 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:17.030479 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:02:17.039044 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.039069 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.043670 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.044704 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.048928 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.054167 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:17.054314 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:17.065683 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.092337 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:17.154546 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.154789 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.158167 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.165048 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.165287 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.167222 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.254765 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.257457 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.260454 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.260623 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.260856 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.264539 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:17.264686 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:17.324846 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.328786 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.334041 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.335361 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.335937 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.336238 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.428031 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.429181 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.433503 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.445796 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.446591 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.448392 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:17.515651 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.532075 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.535056 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:17.535925 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.540351 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.542810 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:17.543682 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:02:17.546958 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.552490 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.565966 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:17.613406 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:17.619779 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.624805 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.627058 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.627530 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.649380 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.650221 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:17.670766 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:17.700162 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.700412 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.703265 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.703878 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.709066 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:17.725798 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:17.727188 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:17.736316 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.740617 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.747856 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:17.754602 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:17.765544 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:02:17.766182 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:17.798866 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.804181 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.804975 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.809074 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.810340 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.821354 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.899647 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.901495 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.908319 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.911193 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:17.914569 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.926127 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.984313 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:17.985033 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:17.989058 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:17.989068 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:17.998003 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.009327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:18.043303 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:18.121693 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:02:18.187747 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.209483 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.209934 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.210073 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.212999 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.222143 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:18.232458 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:18.238952 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.323374 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.326189 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:18.329146 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.334064 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.335441 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.337589 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.366685 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.368422 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.375041 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:18.376729 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 20:02:18.377712 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.381487 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.396517 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:18.421730 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.478700 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.481171 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.485872 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.490764 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.492286 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:18.494390 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.495175 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:18.528344 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:18.556488 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.583168 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.584594 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.589054 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.590231 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.591679 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.593472 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:18.639773 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:18.660482 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:18.687180 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.697686 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.698359 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.698963 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.703852 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:18.705544 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:18.716457 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:18.726758 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.729883 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.731932 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.739423 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:18.753095 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.765514 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.766777 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:18.844254 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.848075 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.849349 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:18.852316 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:18.854569 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.869048 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:18.946610 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:18.952694 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:18.953228 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:18.954572 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:18.954789 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:18.957561 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:18.962805 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:18.975474 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:19.005249 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:19.049530 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.051797 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.053108 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.062592 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.075237 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.081043 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:19.082741 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:19.119427 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.124670 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.127679 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.127869 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.134732 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.146008 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:19.194388 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:19.223750 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.229471 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.229548 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.230051 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.237157 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.242472 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:19.266676 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:19.269407 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 20:02:19.319041 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:19.330639 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.334051 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.341516 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.343073 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.349223 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.349664 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.405426 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.411841 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.416900 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.421059 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.433849 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.435628 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:19.469121 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:19.479022 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:19.483994 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.484395 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.484472 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.489022 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:19.489737 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:19.501835 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.512773 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:19.517663 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:02:19.593807 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:19.598099 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:02:19.598690 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.602218 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.602497 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.604704 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.624403 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:19.668519 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:19.723424 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.723975 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.728153 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.728994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.733505 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.735893 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:19.745993 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:19.824488 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:19.830248 1376 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:19.830344 1376 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:19.830687 1376 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:19.831575 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:19.831652 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:19.831713 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:19.831836 1376 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:02:19.906046 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:19.908102 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:19.908851 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:19.909205 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:19.917052 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:19.926194 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.002315 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.007266 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.008405 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.015280 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.019093 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.026448 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:20.044148 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:20.054998 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.078883 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.081844 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:20.085801 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.091693 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.095958 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.107535 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.108356 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:20.145926 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.160436 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.162237 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.163952 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.169171 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:20.170781 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.172431 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.172572 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:20.181494 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:20.191418 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:20.215834 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.270547 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.272534 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.272992 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.274449 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.281065 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:20.282768 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.283105 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:20.297384 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:20.374811 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.375085 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.380302 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.383500 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.383802 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.391706 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:20.403690 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:20.448406 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:20.449978 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.452065 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.452812 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.455399 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.462198 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.477376 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.520685 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.523715 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.526957 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.526978 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.543165 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.544487 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:20.589776 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:20.605782 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:20.623353 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.629263 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.634089 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.637159 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.641106 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.649170 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.741539 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.745731 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.749948 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.763705 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.765591 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.766834 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.802098 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.803267 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.806602 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.813074 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.817893 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.818498 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:20.909832 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:20.912811 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:20.915673 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:20.919895 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:20.925144 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:20.926297 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:20.934608 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.004191 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:21.015873 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.017803 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.021266 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.022393 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.032378 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.033601 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:21.048007 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:21.061966 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:21.139170 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.139838 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.144472 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.156225 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.156436 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.156556 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.158536 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.173380 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.181324 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.189050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.200335 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:21.209978 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.213065 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.216044 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:21.251981 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:21.278802 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:21.295634 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.299046 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.300813 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.306165 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.306358 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.315026 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.331655 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:21.495097 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.500463 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.501693 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.508606 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.508984 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.509581 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.528434 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:02:21.536044 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:21.538524 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:21.600149 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.605069 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.605373 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.610380 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.616220 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:21.625157 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:21.629932 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.670596 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:21.707763 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.708354 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.713658 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.714995 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.721802 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.725960 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.773576 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:21.821768 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.823978 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.829221 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.831032 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.836889 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.846230 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:21.849009 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.870055 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.879020 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.879725 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.881520 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:21.902165 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:21.945267 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240501 20:02:21.957897 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:21.979904 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:21.984396 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:21.984647 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:21.985051 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:21.993361 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:21.993741 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.004866 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:22.040146 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.084589 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.087829 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.089140 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.089460 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.089579 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.105289 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:22.121243 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:22.187700 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.193226 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.194346 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:22.202358 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:22.207414 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:02:22.208007 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.210897 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.215523 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.253422 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.254911 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.259779 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.265180 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.265709 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.279561 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.357738 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.358327 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.360168 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.361510 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.368459 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:22.370836 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.373234 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:22.450428 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:22.459535 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.469890 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.471294 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.471704 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.472466 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.474874 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.497570 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:22.543036 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.550530 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.552034 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.555927 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.562800 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.563764 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.567744 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:22.593353 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:22.597173 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:22.624476 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.630124 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.633050 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:22.635120 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:22.635700 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:22.642336 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.650257 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.661594 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.737728 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.738690 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:22.738786 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.740173 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.741186 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.749048 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.751403 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:22.784626 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:22.825306 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:22.827757 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:22.842234 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:22.851878 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.852699 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.853483 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.860821 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.860862 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.869378 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.907966 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:22.912043 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:22.913092 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:22.923377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:22.942651 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:22.944857 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:22.971073 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:22.983986 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:23.009524 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.011523 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.016434 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.017398 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.021219 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.022305 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:23.068847 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:23.227286 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:23.231274 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:23.233166 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.233983 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.236251 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.242437 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.246621 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.331346 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.333182 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.334432 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.334645 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.336077 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.341029 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:23.427690 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:02:23.428758 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:23.428876 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:23.432226 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:23.436043 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.437063 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.439157 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.440407 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.441901 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:23.533911 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:23.553414 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:23.556131 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:23.560870 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:23.561522 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:23.562031 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.565451 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.568203 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.570138 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.581207 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.584214 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.591691 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.612139 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.613049 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.618837 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:23.650705 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:23.698866 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.703235 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:23.703979 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.705888 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.708222 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.721132 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:23.767055 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:23.797690 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.797906 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.798310 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.806667 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.809192 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.815150 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:23.865451 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:23.907793 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.909286 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.913628 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.919785 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.923384 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:23.924531 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:23.936275 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.962299 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:23.969230 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:23.969414 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:23.969995 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:23.983119 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:23.985596 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:23.989387 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:24.025313 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:24.036140 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:24.054446 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:24.054592 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:02:24.057855 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.064299 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.064733 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.065958 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.068771 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.073820 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:24.165433 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:24.167295 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:24.168481 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.170634 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.178428 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:24.181320 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.192096 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.290227 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.291471 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.292932 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.297283 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.303655 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.307096 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:24.336604 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.337106 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.350052 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.351092 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.353159 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.353281 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:24.417603 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 20:02:24.419057 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:24.450138 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.450201 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.450428 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.450778 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.462594 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.469025 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:24.470166 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:24.499090 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:24.512693 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:24.545841 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.549180 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.549501 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.553894 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.556067 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.566911 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:24.569531 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:02:24.657167 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.659478 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.661417 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.669246 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:24.678835 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.682948 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.718873 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.721925 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.723810 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.724607 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:24.733834 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.739015 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:24.799093 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:02:24.809731 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:24.857908 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:24.864481 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:24.878453 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:24.935624 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:24.945336 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:24.964938 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:24.965689 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:24.967259 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:24.978556 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.982153 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:24.989074 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:25.008837 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.053211 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.057235 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.063050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:25.064239 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.065261 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.070200 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.090942 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:25.097680 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.163211 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.163866 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.167368 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.167743 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.171950 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.191744 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:25.192127 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:25.290340 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.295845 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.300880 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.302346 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.303419 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.307174 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.316664 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.324070 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.328222 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.338913 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.345588 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.359326 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:25.380060 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:25.383493 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.438026 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.440335 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.441673 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.446852 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:25.454386 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:25.454936 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.457628 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:25.491369 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:25.534245 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.535393 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.540030 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.543104 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.549952 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.550199 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.551367 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:25.552326 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:25.672341 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.674098 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.675057 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.682108 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.683488 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.705739 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.709836 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.714028 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.716401 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.720577 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.735692 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.744663 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:25.784637 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:25.819530 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.822774 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.823220 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.824945 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.831387 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:25.840926 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.929921 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:25.930905 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:25.931672 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:25.934674 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.945016 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:25.945492 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:26.015311 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:26.022068 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:26.044312 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.050747 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.051615 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.056721 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.057614 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.074458 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.075377 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.076684 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.077220 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.093312 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.094309 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:26.097167 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:26.114542 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.118801 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:26.130102 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:26.187754 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:26.197798 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.205700 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.206758 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.208365 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.210242 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.225749 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.241606 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:26.262115 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:26.297708 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.300697 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.305100 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.305472 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.307801 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:26.310863 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.312453 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.414335 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.420230 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.427623 11225 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.427819 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.430155 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.441008 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.441623 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.442919 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.456076 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.463018 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.486272 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.492101 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.551028 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:26.602319 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:26.644891 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:02:26.655175 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:26.664953 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:26.711560 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:26.776219 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.778977 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.781497 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.781726 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.803084 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.808081 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.818089 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.825206 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.826795 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:26.827942 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.828742 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.833282 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.842041 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.924952 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:26.929448 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:26.930389 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:26.938858 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:26.940591 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:26.953465 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:26.961159 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240501 20:02:26.965430 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:27.035049 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.035158 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.041617 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.041874 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.051806 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.057507 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:27.106654 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:27.146962 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.149454 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.155818 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.159054 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.165081 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.169713 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.200587 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.209810 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.210356 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.211033 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.212987 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.217103 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:27.236553 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:27.289953 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:27.308413 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.309991 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.312393 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.314177 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:27.318153 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:27.325495 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.333024 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:27.352331 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:27.364099 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:27.412436 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:27.412892 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.420676 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.422483 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.433207 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.433573 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.434892 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:27.436810 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:02:27.513396 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.513654 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.517030 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.529520 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.539535 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.547171 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:27.552294 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:27.584300 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.588335 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.590750 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.592756 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.607470 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.609860 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:27.637598 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:27.649490 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:27.678084 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:27.678706 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:27.695000 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:27.703270 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.706804 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.711125 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.711278 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.712898 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.716590 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.810012 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.815448 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.818346 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.818346 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.820955 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.828701 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:27.828735 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:02:27.883687 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.884424 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.885164 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.885774 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.887997 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.906529 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:27.944320 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:27.973466 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:27.974000 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:27.974313 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:27.974905 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:27.975203 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:27.975268 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.067627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.067706 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.069911 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.075857 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.081035 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.084113 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:28.125949 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:28.134529 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:28.141615 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:28.178555 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:28.179867 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.187175 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.193792 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.194131 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.225800 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.227548 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.235863 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.245822 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.250182 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.253494 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.263604 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.269150 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:28.314937 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:28.358708 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:28.449052 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:02:28.449673 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:28.495448 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:28.531883 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:28.570035 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.571504 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.572031 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.573428 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.576778 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.578372 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.587035 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.588611 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.592053 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.606357 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.606788 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.634688 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:28.707497 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:28.715693 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:28.717908 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.721635 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.723695 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.729194 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.732187 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:28.733054 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:28.737615 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:28.779951 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:28.823393 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.824046 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.827684 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.846056 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.846410 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.847066 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:28.867285 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:02:28.934794 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:28.950440 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.950845 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.960028 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.961469 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.964612 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.965085 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.977349 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:28.985074 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:28.985785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:28.994199 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:28.997582 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:28.998713 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:29.058785 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:29.095319 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.095927 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.106060 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.106832 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.108634 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.118602 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:29.159512 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:29.169823 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:29.197938 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.201836 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.207012 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.207517 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:29.221357 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.222991 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.230255 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:29.251885 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.316138 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.317057 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.318598 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.320478 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.322978 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.332185 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.387782 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.392364 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.392551 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.394148 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.405167 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.416118 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:29.419093 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:29.432751 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:29.439018 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.496232 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.499238 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.500226 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.503346 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.506111 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.521034 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:29.532778 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 20:02:29.577809 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.608012 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.611939 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.612702 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.613090 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.623745 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.625345 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:29.637490 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:29.653587 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:29.672629 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.703111 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.707047 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.708240 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.708871 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.718333 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.722330 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.769120 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.770514 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.774173 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.776082 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.777814 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:29.785432 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:29.795881 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:29.801882 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:29.874675 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.876708 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.876844 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.877002 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.881093 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.894043 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:29.934412 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:29.976269 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:29.980533 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:29.982612 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:29.983359 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:29.983783 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:29.985787 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:29.991540 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:30.070042 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.074054 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.076817 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:30.081140 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:30.088150 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.101558 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:30.106945 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:30.109840 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:30.150982 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:30.171984 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:30.178340 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 20:02:30.287390 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.287791 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.290210 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.291524 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.294538 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.309197 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:30.342758 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:30.401350 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.401841 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.402233 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.404403 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.405462 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.421943 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.436205 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.453148 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.454159 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.455674 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.470662 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.471446 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.541901 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.543671 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.544466 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.544876 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.545090 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.552268 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:30.617208 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:30.640538 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:30.644757 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:30.649310 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:30.650553 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.654037 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.664891 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.668449 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.671985 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:30.741537 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:30.754660 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.759263 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:30.766955 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:30.769136 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.770730 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.772322 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.778556 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.804116 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.808665 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.817816 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.821890 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.831951 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.844059 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:30.850754 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:30.897357 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:30.902822 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:30.908708 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.911581 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:30.912142 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:30.912403 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:30.999752 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.002957 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.005625 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.017203 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:31.017292 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:31.023388 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:31.031636 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:31.041006 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:31.081284 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:31.084693 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:31.128402 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.130064 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.132373 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.132732 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:31.140565 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.141780 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:31.154165 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:31.165251 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:31.171259 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:31.196377 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.197849 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:31.202158 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.208945 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.209621 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.235507 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.273341 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.277762 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.278165 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.286134 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.286949 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.293206 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:31.303090 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:31.376561 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.376968 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.382015 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.391136 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.394876 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.405931 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:31.438380 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:31.468178 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:31.482216 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.483250 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.486994 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.494786 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.497174 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.517143 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:31.554899 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.557096 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.560349 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.561942 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.563334 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:31.567251 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:31.567651 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:31.591379 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:31.649128 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.651614 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.652585 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.660779 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.673221 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.676029 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:31.691740 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:31.694856 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:31.699224 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:31.826107 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:02:31.865489 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.868657 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.870716 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:31.875304 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.881062 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.889027 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:31.898916 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:31.961125 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:31.963099 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:31.964191 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.965305 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:31.966908 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:31.977236 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:31.992074 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 20:02:31.996984 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:32.056486 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.057842 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.059275 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.060271 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.067243 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.088263 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.089802 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:32.120311 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:32.184212 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.184697 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.188249 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.197036 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.204788 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.207412 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.214820 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:32.227033 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:32.227092 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.234555 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.236990 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.252377 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.265080 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:32.310166 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:32.336395 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:32.340404 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:32.346362 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.348387 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.351276 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.356043 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.358788 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.360044 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:32.370954 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.445823 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:32.446128 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:32.452517 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.452535 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.453161 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.453851 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.472828 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.558151 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.559849 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.560854 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.564749 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.569253 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.579994 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.600898 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:32.604982 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.614375 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.614665 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.615311 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.617808 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.618438 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:32.619591 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.674010 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:02:32.719045 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:32.732930 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.734367 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.736080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.736848 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.737996 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.738956 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:32.788702 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:02:32.844285 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:32.848222 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:32.854212 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.856086 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.867180 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.870100 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.893004 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.917236 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:32.924283 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:32.924816 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:32.926586 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:32.927213 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:32.948326 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.018095 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.018537 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.019752 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.023571 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.023871 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.034303 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:33.057240 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:02:33.121768 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.125896 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.126394 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.128667 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.133710 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:33.135326 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:33.137015 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:33.145117 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:33.239555 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.240401 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.245654 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:33.249848 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:33.256736 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.272614 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.280668 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.281217 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.282583 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.283916 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:33.285529 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:33.287444 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.297322 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.309475 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:33.377662 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:33.377771 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:33.379550 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:33.398093 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.401679 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:33.405141 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:33.406126 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.408535 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.414834 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.431509 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.603853 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.605587 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.608234 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.608948 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.617727 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.626516 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:33.654947 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:33.667954 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 20:02:33.668442 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:33.717286 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.717528 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.717617 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.718745 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:33.720402 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:33.721257 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.721963 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.817341 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.817890 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.820581 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.821911 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.826941 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.827515 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:33.855263 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:33.882963 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:33.886257 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:33.920570 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:33.941318 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:33.942117 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.950417 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.954655 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:33.955713 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:33.955937 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.956802 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:33.959944 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.961279 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:33.973183 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:33.975914 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:33.979637 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.003800 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.087266 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.090219 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.090580 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.097766 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.098433 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.101158 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:34.170167 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:34.185333 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.190055 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.191571 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.191711 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.191764 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.193648 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:34.213398 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.297870 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.299175 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.301524 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.314584 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.325707 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:02:34.328518 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.331840 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.333539 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.336395 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.341145 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.342769 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:34.342929 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.354406 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:34.363019 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:34.365275 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:34.437634 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:34.451122 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.453852 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.454124 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.461088 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.462071 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.482749 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.497925 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:34.559468 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.560837 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.561154 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.569751 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.576553 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.579042 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.648566 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:02:34.667246 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.673542 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.676448 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.678361 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.684180 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:34.690218 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:02:34.690284 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.697734 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.704378 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.724933 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.727895 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.729935 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.733767 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.735982 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.751968 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:34.818764 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:34.832819 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:34.835602 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.838315 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.838639 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.847509 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:34.853308 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.872422 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:34.928268 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.936877 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:34.937438 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:34.937700 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:34.945044 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:34.946501 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:34.948029 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.954141 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:34.956979 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:34.958282 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:34.958860 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:35.051378 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.061302 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.061589 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.064823 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.070188 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.071271 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.102811 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.104271 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.105104 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.111442 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.114547 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.115185 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:35.204732 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:35.278898 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:02:35.301016 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:35.349251 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:35.379546 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:35.427814 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.428061 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.429566 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.433728 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.437350 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.442418 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.449249 13017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.449429 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.454596 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.480710 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.489080 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.493484 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:35.503110 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:35.561399 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.561585 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.562613 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.563385 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.565997 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.584206 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.669345 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.672453 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.672825 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.673223 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:35.678123 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:35.681088 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.696261 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.808203 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.812278 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.812496 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.812496 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.821581 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.824579 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.826488 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.830942 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.833285 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:35.853669 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:35.854097 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.860810 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.868451 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:35.878064 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:35.878502 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:35.917467 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:35.953208 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:35.953377 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:35.954515 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:35.957852 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:35.961061 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:35.962440 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.001003 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:36.051649 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.052557 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.059561 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.062255 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.074542 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:36.074944 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:36.076170 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:36.093027 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.116571 8081 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:36.116668 8081 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:36.117026 8081 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:36.118018 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:36.118501 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:36.118557 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:36.118746 8081 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:02:36.163831 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.165082 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.171010 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.173024 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.174409 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.189177 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.232189 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:36.236784 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:36.237892 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.238132 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.238718 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.240585 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.241473 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:36.284183 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:36.328512 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:36.337141 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.337565 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.341344 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.353122 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.353514 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.355938 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.356339 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:36.426982 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:36.427150 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:36.451180 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.452600 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.456833 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.464565 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.479880 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.484627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.519701 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:36.523217 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.527051 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.530660 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.535295 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.536525 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.553650 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:36.554828 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.611340 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.616264 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.616531 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.622820 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.623950 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.637277 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:36.662137 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:36.720602 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.724611 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.726295 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.728632 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.728686 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.745316 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.805091 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:36.806766 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:36.841105 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.854041 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.854884 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.855589 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.881587 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:36.889124 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.891578 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.892333 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:36.904150 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:36.906888 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:36.908746 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:36.913060 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:36.950482 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:02:37.004933 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240501 20:02:37.025182 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:37.030366 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:37.111289 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:37.154803 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:37.185256 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.194756 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.195271 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.198006 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.198465 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.207893 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:37.251132 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:37.263549 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.266520 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.268869 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.273134 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.286859 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:37.287890 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:37.297190 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:37.306524 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:37.330039 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:37.344710 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.345482 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.350018 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.351202 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.353050 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.369676 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:37.380129 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:37.450695 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.453910 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.455948 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.458397 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.465662 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.481531 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:37.547612 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:37.568429 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.569542 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.580515 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.581720 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.604753 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.608762 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.613031 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.616284 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.619064 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:37.620743 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:37.630534 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.649281 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.658044 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.729728 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.730393 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.731696 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.737052 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.740053 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.740804 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:37.816257 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:37.816951 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:37.819005 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:02:37.831529 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.834977 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.836244 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.840173 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.846273 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.846776 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:37.874970 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:37.958200 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.959144 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.961124 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.968694 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:37.973942 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.977468 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:37.983603 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:37.985566 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:37.985816 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:37.986470 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.001639 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.015692 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:38.074842 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:38.087739 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:38.096956 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:38.108713 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.110376 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.118430 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:38.118718 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:38.121913 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.126559 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.136214 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:38.177603 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:38.216022 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.216575 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.222397 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.227897 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.230675 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.234066 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.320992 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.322351 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.329967 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.330639 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.347404 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.358667 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:38.375202 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:38.379981 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.383097 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.384491 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.386370 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.392850 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:38.394204 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:38.419827 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:38.425012 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:38.491662 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.492475 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.496829 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.501169 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.504793 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.515741 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:38.520555 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:38.543641 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:38.590387 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:38.593137 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.596230 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:38.601799 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:38.604238 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.605401 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.617722 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.621857 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.685007 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.685739 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.689388 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.693768 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.705045 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.709034 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:38.725124 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 20:02:38.725776 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:38.749845 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:38.913446 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:38.913969 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:38.918480 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:38.929988 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:38.934708 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:38.941565 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:38.945245 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:38.951557 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:38.969024 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:39.027285 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.030195 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.030809 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.033026 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.040702 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.045055 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.054572 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:39.057070 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.070075 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.075223 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:39.076171 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:39.080113 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.094201 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.094538 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.183025 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.188161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.190135 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.193779 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.195123 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.207252 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.305640 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.308682 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.308753 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.308785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.309618 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.322785 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:39.336290 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:02:39.346850 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:39.421730 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.423630 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.427584 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.438227 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.442796 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.448208 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.448343 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.453009 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.453202 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.469102 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.479249 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.488536 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:39.494818 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:39.564059 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:39.574575 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.574632 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.574689 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.577101 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.580008 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:39.581202 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.594125 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.600301 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:39.610797 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:39.626157 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:39.682605 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.685348 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.686623 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.686913 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.689622 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.690501 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.694319 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:39.732299 11050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240501 20:02:39.789146 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.790205 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.791499 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.792766 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.799788 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.824203 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.833935 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:39.844760 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.850499 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.857972 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.858773 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.865190 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.865686 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:39.886910 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:39.954895 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:39.957934 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:39.957975 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:39.966611 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:39.974695 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:39.986056 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:40.004575 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:40.024067 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:40.065431 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.065814 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.070858 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.077055 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.086441 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.093132 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.165622 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.166029 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:40.176685 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:40.177225 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.180030 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.180209 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:40.195719 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:40.214715 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.225333 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.226333 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.234777 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.234987 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.237510 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.243439 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:40.255437 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:40.320308 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:40.332564 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.334719 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.343539 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.343884 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.346972 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.356622 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.522035 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:40.523728 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:02:40.524706 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.530954 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.533861 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.537891 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:40.545992 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:40.553336 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:40.557523 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:40.606206 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:40.641508 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.646070 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.647058 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.647503 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:40.647562 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:40.649704 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.656936 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:40.725157 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:40.731099 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:40.742417 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.747062 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.748380 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.750140 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.755923 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.765801 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:40.780167 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:40.797122 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:40.852689 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.856825 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.865226 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.870509 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.871582 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.878916 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:40.902364 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:40.906179 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.910570 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:40.923945 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:40.926764 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:40.933616 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.014539 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.015425 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.017117 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.018155 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.019347 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.033965 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:41.034471 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:41.117712 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.118700 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.119277 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.121855 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.122048 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.124310 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:41.124487 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:41.133137 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:41.186064 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:41.194904 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:41.230635 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.243081 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.244729 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.245605 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.246050 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.247056 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.249814 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.255048 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.259337 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.265173 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.270609 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.304369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:41.315210 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:41.329195 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:41.365204 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:41.384258 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.387697 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.389544 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.390067 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.390300 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.391557 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.487865 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.489953 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.492204 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.492247 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.501385 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.504499 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:41.583762 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:41.597877 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.598541 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.602941 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.617307 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.620874 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.621462 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.644078 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.645318 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.651036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.654783 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.666224 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.666455 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:41.712823 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:41.714207 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:41.763010 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.764488 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.766317 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.766607 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.771970 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.773706 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.863870 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:41.864382 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.871330 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.873452 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.873664 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:41.881839 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:41.903316 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:02:41.968513 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:41.969179 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:41.970252 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.971086 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:41.971323 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:41.978646 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:41.986493 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.031339 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.033321 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.033627 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.037012 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.043846 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.050954 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240501 20:02:42.053721 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.062759 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:42.084764 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:42.100522 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:42.102331 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:42.276567 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:42.280709 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.281257 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.283177 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.285148 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.285569 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.306911 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.309106 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:42.316993 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:42.342491 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:42.375612 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.376296 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.376555 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.377616 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.381753 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.381994 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.471427 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.472146 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.473651 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.477475 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.485396 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.489821 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:42.527534 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:42.585548 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.586282 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.587735 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.591534 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.601258 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:42.610100 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:42.621148 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.647599 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.647948 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.651108 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.657402 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.659411 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:42.660372 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.669099 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.675341 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:42.687664 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:42.745397 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.746840 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.750989 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.760563 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.761056 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.780102 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:42.781622 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:42.849061 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.856007 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.857515 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.870211 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.872208 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.883777 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:42.887140 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:42.892007 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:42.924520 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:02:42.943610 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:42.970775 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:42.977347 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:42.977900 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:42.978298 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:42.981397 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:42.998970 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.008399 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.011840 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.015903 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.018488 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.028764 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.040654 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:43.056453 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:43.109756 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:43.130620 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.131991 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.135775 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.136474 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.143587 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.155141 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.240422 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.241525 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.243588 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.244999 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.245740 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.249703 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:43.277081 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:43.291445 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:43.295917 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:43.334589 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.340559 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.353083 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.353226 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.354455 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:43.378520 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:43.389034 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:43.404708 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:43.418793 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:43.419032 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:43.421835 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.425094 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.425827 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.437650 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.454413 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:43.477965 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:43.527781 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:43.532498 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.536176 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.536613 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.536993 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.538861 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.546350 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:43.568974 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:43.578752 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:43.628224 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.629418 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.631626 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.645298 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.649515 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.653929 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:43.655608 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:43.746526 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.753307 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.754873 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.755844 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:43.756711 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:43.766276 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 20:02:43.770270 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:43.783478 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.790059 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:43.792642 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.794986 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:43.795296 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:43.803217 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:43.817332 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:43.872357 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:44.026741 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:44.088480 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.090008 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.100522 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.100869 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.104146 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.126467 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.137276 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.140816 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.141836 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.144583 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.172523 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.181623 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:44.194166 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:44.248081 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.248461 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.248811 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.253705 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.257079 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.262339 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.344323 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.347208 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.350085 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.357031 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.357414 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.358812 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:44.375734 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:44.376499 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:44.402031 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:02:44.471143 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:44.471292 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.477653 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.480190 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.486985 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.489663 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.491441 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.493362 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.509819 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.519511 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.525260 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.541587 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.542212 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:44.575809 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:44.602968 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.605777 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.609714 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.613615 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.619403 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.625344 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:44.639109 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:44.641901 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:44.710084 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:44.719489 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.720347 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.722079 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.724251 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.726742 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.738312 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:44.758361 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 20:02:44.793967 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:44.798767 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:44.833642 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.834875 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.837728 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.847802 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:44.848345 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:44.857889 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.860965 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.881295 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.881906 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.891561 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:44.892122 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.894177 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.901470 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:44.905627 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:44.920015 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:44.993893 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:44.994974 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:44.996994 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:44.997613 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:44.998189 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.003127 12109 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:02:45.003225 12109 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:45.003582 12109 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:02:45.005664 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:45.010488 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:45.010561 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:02:45.010757 12109 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:02:45.017964 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:45.051949 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:45.053160 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:45.103747 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.103819 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.108327 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.117703 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.119378 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.124244 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.226598 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.227303 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.232687 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.234302 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.234719 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.235412 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.239529 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.248639 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.250960 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.261822 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.262212 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.282639 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:45.289904 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:45.351929 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:45.367637 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.372071 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.374109 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.380613 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.383085 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.384845 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:45.392652 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:45.443290 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:45.457605 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:02:45.476404 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.480877 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.483047 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.484313 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.485152 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.485942 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:45.493834 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:45.578151 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:45.579339 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:45.580617 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.583083 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.585151 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.585723 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.593855 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:45.704981 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:45.736691 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:45.772368 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:45.783710 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.785006 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.789659 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.798297 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.802228 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.805536 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:45.823163 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:45.914139 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.915344 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.915961 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.925693 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.932236 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:45.946342 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:45.947008 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.947702 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.947788 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:45.948112 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:45.949391 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:45.950882 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:45.989806 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:46.010313 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:46.045049 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:46.060135 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.062712 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.063473 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.066924 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.071491 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.073866 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:46.090812 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:46.163684 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:46.168967 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:46.169238 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.170361 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.171728 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.174623 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.177366 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.193138 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:46.252722 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:46.278584 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.282095 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.285060 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.291690 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.299013 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.302253 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.303830 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.305886 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:46.324782 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:46.329339 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.332291 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.335280 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.347832 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:46.381965 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:46.434664 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:46.439633 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.441170 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.444164 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.445084 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.458604 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.459757 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.545576 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.546861 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.547209 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.555811 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.556269 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.556459 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:46.629458 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:46.661125 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.676205 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.680698 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.683436 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.691663 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.692057 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.692740 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.709782 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.713367 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.716658 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.721194 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.727236 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:46.741931 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:46.810833 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:46.825999 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.828713 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:46.829536 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.837062 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.839179 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.855374 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:46.879222 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:46.933946 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:46.937031 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:46.937435 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.940994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:46.943835 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:46.952955 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:46.981267 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:47.019771 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:47.035275 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:02:47.048498 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.049166 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.054706 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.060429 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.061054 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.062846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.071979 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.074554 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.076926 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.093456 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.095990 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.124967 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:47.147461 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:47.150224 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:47.152016 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:47.160535 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:02:47.213214 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.215036 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.217206 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.217960 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.218230 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.227203 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:47.324182 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:47.358830 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:47.372571 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:47.406991 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.414357 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.415720 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.421604 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.422925 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.435055 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:47.497112 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:47.522873 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.526481 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.528666 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.534842 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.535586 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.544027 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:47.606724 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:47.627871 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.628634 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.629278 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.638896 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.639833 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.645591 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:47.723975 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:47.749274 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.755965 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.756111 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.762174 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.764312 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.764961 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.774827 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.794200 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.795254 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.795282 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.801671 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.803179 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:47.805295 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:47.848158 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:47.900141 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.900439 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.904166 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.906924 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.914234 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.916918 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:47.994870 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:47.996429 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:47.997691 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:47.997737 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:47.999934 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.006085 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:48.025287 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:02:48.053154 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.101083 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.114593 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.122293 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.130060 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.131402 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:48.132052 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:48.141742 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:48.145851 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.148100 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.151935 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:48.162254 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:48.175559 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:48.179616 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.181222 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.181856 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.184064 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.272192 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.272591 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.273918 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.275012 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:48.283341 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.283587 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.294016 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:48.305359 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:48.318722 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:48.330575 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:48.331148 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.371811 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.372380 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.374714 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.377388 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.383219 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.395562 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:48.407151 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:48.473307 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.473999 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.485095 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.485829 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.492563 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.493638 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:48.502509 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.531869 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.538266 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.539005 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.539672 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.543180 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.545540 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:48.555809 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:48.633492 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.635215 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.642292 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.642504 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.644534 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.664588 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:48.698071 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:48.708137 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.742024 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.745333 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.747031 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.749171 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.749514 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.774410 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:48.806427 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 20:02:48.806567 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.821923 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.828284 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.828960 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.830487 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.832680 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:48.840032 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:48.850672 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:48.850714 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.901645 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:48.904918 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:48.906715 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:48.907133 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:48.916074 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:48.916654 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:49.049016 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:49.065747 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:49.131004 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:49.141824 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.144047 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.144716 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.152729 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.161079 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.174086 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.225488 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.230890 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.232686 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.234987 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.245072 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:49.252123 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:49.253369 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.334436 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.334674 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.337504 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.338819 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.346424 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.347679 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:49.429754 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:49.449764 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.451084 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.473337 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.484977 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.485527 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.495617 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.497979 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.501039 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.503877 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.505246 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:49.505738 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:49.513424 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.517446 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:49.547547 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:49.593283 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:49.604216 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.608134 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.609684 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.612496 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.615850 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.629554 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:49.662369 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:49.708593 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:02:49.715646 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.718004 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.719333 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.721882 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.726477 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.727176 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:49.742152 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:49.746225 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:49.753866 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:49.827761 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.827785 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.828756 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.832684 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.838325 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.853312 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.872975 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:49.876667 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:49.881484 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:49.887248 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:02:49.888662 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.890043 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:49.892627 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.907182 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.986231 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.987175 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:49.988337 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:49.989233 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:49.990026 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:49.990783 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:50.000404 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:50.041743 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:50.092478 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.093747 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.096122 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.101907 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.110273 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.115808 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:50.134627 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240501 20:02:50.202628 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.213183 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.214906 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.215092 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.221093 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.244616 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.258042 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.258671 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.267957 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:50.270969 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:50.279614 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.283813 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.284304 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:50.359609 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:50.365952 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.366796 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.367480 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.369887 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.374254 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:50.378291 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:50.380540 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.462193 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.465955 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.468156 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.470315 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:50.485157 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:50.486613 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.492383 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:50.505625 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:50.560158 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.560324 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.560963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.572961 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.578987 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.606577 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.606781 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.609822 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.615378 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.615428 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.627353 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.637777 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:50.718317 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:50.720698 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:50.729523 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:50.754079 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:50.756191 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:50.782058 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:02:50.783581 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:50.798185 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:50.867163 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:50.909739 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:50.950484 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.952555 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:50.958393 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.962708 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.963492 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:50.963546 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:50.969429 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:50.977327 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.992223 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:50.996657 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.008839 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.009658 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.094942 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.094942 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.096827 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.102295 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.102499 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.107252 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:51.137477 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:51.197875 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.201747 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.204994 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.214083 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.214862 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.227996 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:51.231307 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:51.281457 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:51.319036 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:51.319595 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.329843 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.330587 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.331729 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.332252 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.351770 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.352658 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.355203 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:51.366439 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:51.373756 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.373819 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:51.377979 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:51.381058 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.389398 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.468250 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.472702 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.472846 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.472926 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.472986 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.474278 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:51.504637 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:51.517617 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:51.527379 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:51.528441 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:51.569202 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.569700 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.571197 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.578969 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.584009 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.585813 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.680033 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.688930 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.691838 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.694283 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.699110 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.700460 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.728632 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.729337 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.735344 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:51.746294 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:51.746608 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.747176 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.751363 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.833990 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.836902 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.839076 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.843216 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:51.854903 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.855623 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:51.925062 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:51.941268 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:51.942552 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.943351 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:51.943974 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:51.945818 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:51.954373 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:52.024019 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:52.040000 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.043371 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.048050 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.053180 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:52.062359 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:52.067566 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.068249 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:52.090236 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:52.099512 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:02:52.106336 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.112974 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.113577 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.115069 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:52.124259 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:52.128398 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:52.129313 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:52.188206 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240501 20:02:52.207888 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.208703 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.210156 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.213595 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.224687 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.226667 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:52.274016 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:52.319559 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.320165 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.326690 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.331149 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.334933 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.338161 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:52.339550 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:52.341013 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:52.370143 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:52.420070 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.423086 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.423316 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.426645 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.430500 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.435711 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:52.513586 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:52.533035 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:52.623493 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.628006 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.632387 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.633363 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.637854 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.638271 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:52.747567 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.749246 12752 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.749380 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.760185 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.764478 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:52.764508 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.773066 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:52.777626 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:52.785068 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.793116 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.807340 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.816517 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:52.856307 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:52.900415 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.902024 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:52.902004 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:52.902138 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:52.902556 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:52.925788 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:52.928453 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:52.957983 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:53.004060 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.006726 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.007203 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.007920 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.010948 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.010965 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.035252 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.041364 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:53.044730 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:53.068460 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:02:53.080641 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:53.117332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.118252 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.125578 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.132643 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.141415 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.145370 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.175264 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.175681 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.176857 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.178433 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.179296 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.185304 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.205958 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:53.231336 12973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:53.231994 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.289777 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.290033 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.290414 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.291210 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.301357 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.305719 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.316857 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.394084 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.395298 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.396257 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.410492 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.418746 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.421056 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.470633 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.482379 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.487524 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.489244 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.490602 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.502627 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.505553 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.513485 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:53.554092 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.557907 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.559454 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:53.563093 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:53.563681 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.572108 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.572666 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.626009 12709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:53.626751 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.668160 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.668375 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.670326 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.669781 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.681641 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.682644 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:53.711602 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:53.713953 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.771869 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.772468 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.776697 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.777487 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.790689 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.799597 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:53.801153 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:53.854629 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.854755 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.858399 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.870193 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:53.875273 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:02:53.881925 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.889959 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.921022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:53.924537 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:53.928088 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:53.928099 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:53.928687 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:53.943198 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.021169 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.026250 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.026895 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.027957 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.035815 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.044736 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:54.058082 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:54.084844 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:54.094240 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:02:54.102464 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:54.121639 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:54.174082 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.176036 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.181077 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.181286 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.183387 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.189044 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:54.304576 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:54.326864 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.328037 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.330685 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.330976 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.333639 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.355463 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:54.411582 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:54.436681 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.440768 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.442278 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.442308 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.450382 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.457136 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:54.491348 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:54.509325 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:54.521258 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.522621 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.522866 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.526247 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.534225 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.549386 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.588160 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.596719 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.597436 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.597739 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.598637 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.599499 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:54.621863 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:54.649993 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:54.700088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.703979 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.706838 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.708992 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:54.720930 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:54.725069 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.726677 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.809638 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.814813 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.817516 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.818226 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.826912 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.828096 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:54.833642 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:54.882725 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.885741 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.885794 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.894239 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.894773 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.901094 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:54.944254 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:54.945341 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:54.974563 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:54.975392 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:54.977819 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:54.978219 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:54.979126 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:54.995754 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.053346 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:55.076272 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.077965 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.078917 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.079982 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:55.097087 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:55.101796 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:55.106302 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.113322 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.125712 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:55.201987 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.203761 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.209031 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:55.217626 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:55.223541 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.232743 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.236085 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.237188 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.238521 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.240029 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.247313 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.253137 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.261016 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:55.293627 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:02:55.360370 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.360985 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.361166 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.362202 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.363374 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.369048 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.370884 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:55.385577 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:02:55.464881 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.465804 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.468113 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.469380 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.489495 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.494763 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.509162 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:55.541949 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:55.563300 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.571166 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.582973 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.585115 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:55.596143 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:55.597656 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.608577 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.611970 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.617949 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.627466 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.629595 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:02:55.637503 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.645972 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.646232 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.695006 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:55.735378 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.735664 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:55.738181 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.739866 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.747733 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.753474 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:55.786397 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:55.825668 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:55.845495 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:02:55.932348 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:55.936234 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:55.938421 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.939536 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:55.940541 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:55.945577 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:55.953068 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:02:55.986721 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.044953 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.045166 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.047453 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.048422 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.050041 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.076239 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:56.111975 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:56.155686 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.157276 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.158118 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:56.159153 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.160971 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.166122 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.167227 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.174723 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:56.181507 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.274880 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.279961 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.291491 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.292562 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.297044 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.297173 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:56.298930 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.303292 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.307897 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.320729 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.323292 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.331059 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.344658 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.430917 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.431813 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.431959 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.435811 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:56.442957 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:56.447048 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.449859 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.525626 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.528153 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.530200 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.532334 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.535277 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.558064 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:56.644582 11049 consensus_peers.cc:485] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:56.648399 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:56.674398 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.674604 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.676299 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:02:56.677156 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:56.677608 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.678153 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.687927 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.690202 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.697057 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:56.699564 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.700644 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.724287 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.731906 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.733944 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:56.746469 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:56.811406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.811816 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.817744 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:56.826836 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:56.828459 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.830889 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.831220 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:56.838965 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:56.919569 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:56.921603 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:56.925817 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:56.927428 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:56.928035 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:56.945094 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:57.015746 11311 consensus_peers.cc:485] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:02:57.029992 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.040148 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.044312 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.044963 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.045789 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.046231 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.058817 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.068676 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.080598 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.081687 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.101509 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.116426 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:57.192023 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240501 20:02:57.197631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.200490 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.203481 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.204769 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.205335 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.211102 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:57.214149 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:02:57.273301 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.275964 11312 consensus_peers.cc:485] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.298539 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:57.303242 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.303263 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.304908 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.305802 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.307842 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.313192 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:57.336767 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:57.388017 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:57.415468 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.417515 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.422719 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.426653 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.430465 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.434967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.454455 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.459125 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.464134 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.465077 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.467931 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.489369 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:57.497754 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.714498 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:57.717543 12972 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.745908 10919 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.763231 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:57.767629 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.769378 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.773970 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.782374 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.800165 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.802784 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.815486 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.823741 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.827314 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.829190 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:57.836441 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.837314 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:57.855931 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:57.863632 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:02:57.922145 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:57.923861 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:57.927129 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:57.928277 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.933432 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:57.943748 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:57.977998 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:02:58.031407 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.033303 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.034435 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.034685 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.035327 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.042254 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:58.132337 11311 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:02:58.133255 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.138100 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.144484 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.145056 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.150839 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:58.161790 12972 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:02:58.164186 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.205018 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.207839 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.208438 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.212061 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:58.217909 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:02:58.227424 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.233309 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:02:58.247051 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:02:58.267692 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:02:58.268373 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:58.307883 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.308478 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.313751 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.318161 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.325915 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.326143 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.397578 18594 raft_consensus.cc:998] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.040s)
I20240501 20:02:58.397924 18594 consensus_queue.cc:237] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:58.399601 12773 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.400553 18679 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:58.401082 11376 raft_consensus.cc:1261] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.401724 17286 consensus_queue.cc:1035] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:02:58.402918 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:58.403820 18679 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.405480 12773 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.406508 10839 catalog_manager.cc:5484] T b4f0699b5bdf44499c87ff1541cd2ad3 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.409288 11376 raft_consensus.cc:2932] T b4f0699b5bdf44499c87ff1541cd2ad3 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.415416 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.417321 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.421219 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.424017 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:58.439080 11311 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:58.442653 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:58.444466 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:58.446126 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:58.450145 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:02:58.462937 12973 consensus_peers.cc:485] T a170766f497f491ba9b45f92f2a0e751 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:58.493443 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:58.510794 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.513351 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.516083 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.525514 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.529093 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.540534 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.577716 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.579094 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.586333 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.586557 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.586805 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.597376 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.658581 18508 raft_consensus.cc:998] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.189s)
I20240501 20:02:58.658900 18508 consensus_queue.cc:237] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:02:58.660115 11244 raft_consensus.cc:1261] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.660526 18804 consensus_queue.cc:1035] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:02:58.661633 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:58.662998 18804 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:58.663107 11244 raft_consensus.cc:2932] T 8fc8c5aab87341e3b79a8e121e6db01e P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:58.665457 10839 catalog_manager.cc:5484] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:58.691162 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.691749 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.694694 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.695531 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.697114 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.705121 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.713549 16430 raft_consensus.cc:998] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.288s)
I20240501 20:02:58.713863 16430 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 20:02:58.716001 11244 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:02:58.716439 16430 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:58.721922 16430 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.723506 11244 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
W20240501 20:02:58.726944 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:58.726959 10840 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.734944 13035 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:58.736088 11244 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:02:58.736418 16430 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:02:58.737792 16430 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:58.739195 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 19f79cd7d4ec4b4697a2637e6d14a892 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:58.739411 11244 raft_consensus.cc:2932] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:58.739867 10838 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 9 to 10, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
W20240501 20:02:58.740157 12973 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): Couldn't send request to peer 1679f84270414f67ac67c882220e1472. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19f79cd7d4ec4b4697a2637e6d14a892. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:58.748720 18829 raft_consensus.cc:998] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.168s)
I20240501 20:02:58.749017 18829 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:02:58.750200 10983 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:02:58.750293 11244 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer e3c6b84864cb4358a9d7e0f2a65ba83d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:02:58.750629 18829 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:02:58.750939 18829 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:02:58.752343 18840 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:58.753132 10983 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:58.753245 11244 raft_consensus.cc:2932] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:58.755154 10838 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 9 to 10, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 2 leader_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:58.762619 18847 raft_consensus.cc:998] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.127s)
W20240501 20:02:58.762872 12708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:02:58.762957 18847 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:58.764144 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:02:58.764271 11376 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:02:58.764613 18847 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:02:58.764735 18849 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:02:58.766441 18847 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.767050 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.768574 10832 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 6 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.768949 18271 raft_consensus.cc:998] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.115s)
I20240501 20:02:58.769237 18271 consensus_queue.cc:237] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:02:58.770216 11244 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:02:58.770601 18852 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:02:58.770874 11376 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.773473 18852 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:58.774706 11244 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:58.775606 10840 catalog_manager.cc:5484] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 5 to 6, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:58.781932 10983 consensus_queue.cc:237] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
W20240501 20:02:58.783177 10918 consensus_peers.cc:485] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e48d711c5714ee58ebbcbd094ebf3e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:58.783560 11244 raft_consensus.cc:1261] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.783963 18852 consensus_queue.cc:1035] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:58.786305 18271 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.786564 11244 raft_consensus.cc:2932] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.787585 10828 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8e48d711c5714ee58ebbcbd094ebf3e7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:58.788661 10840 catalog_manager.cc:5484] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 6 to 7, VOTER e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
W20240501 20:02:58.805896 11312 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:58.827719 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.828265 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.829385 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.839183 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.839619 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.840220 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.857239 18594 raft_consensus.cc:998] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.389s)
I20240501 20:02:58.857589 18594 consensus_queue.cc:237] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 20:02:58.859361 13035 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.859799 18679 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:58.862623 10983 raft_consensus.cc:1261] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:58.863018 18679 consensus_queue.cc:1035] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:02:58.865768 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:02:58.876029 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:02:58.878086 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240501 20:02:58.887264 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:58.891042 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:58.891644 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.895220 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:58.896680 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:58.897221 18679 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.899931 13035 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.903100 10839 catalog_manager.cc:5484] T 269047cc3a504d7488747166efce71fc P e3c6b84864cb4358a9d7e0f2a65ba83d reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.904042 10983 raft_consensus.cc:2932] T 269047cc3a504d7488747166efce71fc P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:58.910346 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:58.959059 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:58.960026 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:02:58.960474 18849 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:58.961865 12619 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.963399 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.964207 10837 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:58.971401 11354 tablet_service.cc:1514] Processing DeleteTablet for tablet ee298406a4824f1aa2fa742b7dec1026 with delete_type TABLET_DATA_TOMBSTONED (TS 1679f84270414f67ac67c882220e1472 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42722
I20240501 20:02:58.971758 18970 tablet_replica.cc:330] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: stopping tablet replica
I20240501 20:02:58.971907 18970 raft_consensus.cc:2229] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:02:58.972033 18970 raft_consensus.cc:2258] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:02:58.972617 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:02:58.973086 18970 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:02:58.974172 10983 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:02:58.974578 12619 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:02:58.976711 12619 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:58.977293 18970 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 20:02:58.977241 10983 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:58.977357 18970 log.cc:1196] T ee298406a4824f1aa2fa742b7dec1026 P 1679f84270414f67ac67c882220e1472: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee298406a4824f1aa2fa742b7dec1026
I20240501 20:02:58.977916 10828 catalog_manager.cc:4830] TS 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): tablet ee298406a4824f1aa2fa742b7dec1026 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
W20240501 20:02:58.978710 12707 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555): Couldn't send request to peer 4456bc4917c644f8b69b4f5bfb8e1166. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:58.978726 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ee298406a4824f1aa2fa742b7dec1026 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:58.979070 10837 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 9 to 10, VOTER 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:59.007104 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:59.008167 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.009119 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.019312 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.021870 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.022966 18804 raft_consensus.cc:998] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.309s)
I20240501 20:02:59.023245 18804 consensus_queue.cc:237] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
W20240501 20:02:59.025671 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.026144 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.028139 11244 raft_consensus.cc:1261] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:59.028488 18508 consensus_queue.cc:1035] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:59.037417 18508 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:59.039801 11244 raft_consensus.cc:2932] T 5d032fb77ac944629a45b808cc76c2f1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:59.040074 10836 catalog_manager.cc:5484] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } } }
I20240501 20:02:59.045672 18804 raft_consensus.cc:998] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.417s)
I20240501 20:02:59.045974 18804 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.047559 11114 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.047747 13035 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.048246 18803 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.048481 18508 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.050525 18803 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.050962 13035 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.050962 11114 raft_consensus.cc:2932] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.053328 10834 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 8 to 9, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.057904 18801 raft_consensus.cc:998] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.407s)
I20240501 20:02:59.058251 18801 consensus_queue.cc:237] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:59.059670 12773 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:59.060031 18803 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:59.060168 11114 raft_consensus.cc:1261] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:59.060490 18803 consensus_queue.cc:1035] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:59.061815 18801 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.062278 12773 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.063820 10836 catalog_manager.cc:5484] T c3ee13ec9bb1468f90cfc184455f9fff P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.064380 11114 raft_consensus.cc:2932] T c3ee13ec9bb1468f90cfc184455f9fff P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.139750 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:59.142720 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.143098 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.145761 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.146448 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.146850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.153285 18852 raft_consensus.cc:998] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.021s)
I20240501 20:02:59.153594 18852 consensus_queue.cc:237] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:02:59.156054 11244 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.156627 13035 raft_consensus.cc:1261] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.156991 18859 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.158623 18859 consensus_queue.cc:1035] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.169675 18859 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:59.172685 11244 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:59.174582 13035 raft_consensus.cc:2932] T 9fcd6776aebf47c99f05850de83b36e6 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:02:59.177469 10836 catalog_manager.cc:5484] T 9fcd6776aebf47c99f05850de83b36e6 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
W20240501 20:02:59.180291 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:59.206645 11376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7"
dest_uuid: "1679f84270414f67ac67c882220e1472"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42774
I20240501 20:02:59.206758 11376 raft_consensus.cc:595] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:02:59.229425 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.295s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:02:59.231004 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:59.247234 17286 raft_consensus.cc:998] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Attempting to remove follower ea26a64181784350bcd3540d0633b219 from the Raft config. Reason: Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.409s)
I20240501 20:02:59.247664 17286 consensus_queue.cc:237] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } }
I20240501 20:02:59.249287 13035 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:59.249318 11376 raft_consensus.cc:1261] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:02:59.249711 18594 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:59.249934 18679 consensus_queue.cc:1035] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:02:59.255049 18679 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:59.257853 13035 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P e3c6b84864cb4358a9d7e0f2a65ba83d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:59.259673 11376 raft_consensus.cc:2932] T 219742bd987e4a1990233fa8c972c232 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
I20240501 20:02:59.260392 10841 catalog_manager.cc:5484] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 6 to 7, VOTER ea26a64181784350bcd3540d0633b219 (127.30.1.71) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } }
W20240501 20:02:59.268926 12972 consensus_peers.cc:462] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:02:59.270145 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:59.271730 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.283361 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.290894 19141 ts_tablet_manager.cc:902] T 8e48d711c5714ee58ebbcbd094ebf3e7 P e3c6b84864cb4358a9d7e0f2a65ba83d: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
F20240501 20:02:59.291061 19141 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.queue_time_us, tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.queue_time_us) (tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.run_cpu_time_us, tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.run_cpu_time_us) (tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.run_wall_time_us, tablet-download-pool-0a1bbe74dbc74fa687b0a64d3aa203f3.run_wall_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.queue_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.queue_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_cpu_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_cpu_time_us) (tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_wall_time_us, tablet-download-pool-156d2ba7ff2c403db2f63cbae4acfec1.run_wall_time_us) (tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.queue_time_us, tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.queue_time_us) (tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.run_cpu_time_us, tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.run_cpu_time_us) (tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.run_wall_time_us, tablet-download-pool-19bfb955b9c94fdb8e0381e5b1855248.run_wall_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.queue_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.queue_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_cpu_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_cpu_time_us) (tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_wall_time_us, tablet-download-pool-19d8d7d255494a49920849d10b800fa7.run_wall_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.queue_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.queue_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_cpu_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_cpu_time_us) (tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_wall_time_us, tablet-download-pool-19f79cd7d4ec4b4697a2637e6d14a892.run_wall_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.queue_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.queue_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_cpu_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_cpu_time_us) (tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_wall_time_us, tablet-download-pool-219742bd987e4a1990233fa8c972c232.run_wall_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.queue_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.queue_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.run_cpu_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.run_cpu_time_us) (tablet-download-pool-269047cc3a504d7488747166efce71fc.run_wall_time_us, tablet-download-pool-269047cc3a504d7488747166efce71fc.run_wall_time_us) (tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.queue_time_us, tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.queue_time_us) (tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.run_cpu_time_us, tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.run_cpu_time_us) (tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.run_wall_time_us, tablet-download-pool-2def1cda07c349c69b3abe25804d3d1d.run_wall_time_us) (tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.queue_time_us, tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.queue_time_us) (tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.run_cpu_time_us, tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.run_cpu_time_us) (tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.run_wall_time_us, tablet-download-pool-54d2db0f85ac4a90a1df779f89398463.run_wall_time_us) (tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.queue_time_us, tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.queue_time_us) (tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.run_cpu_time_us, tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.run_cpu_time_us) (tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.run_wall_time_us, tablet-download-pool-5eb1f0dcd4cf49cd969eb28f1eb5d9ea.run_wall_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.queue_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_cpu_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_cpu_time_us) (tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_wall_time_us, tablet-download-pool-6fd958b083bb430a99130a18f91acdf7.run_wall_time_us) (tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.queue_time_us, tablet-download-pool-8e48d711c5714ee58ebbcbd094ebf3e7.queue_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.queue_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.queue_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_cpu_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_cpu_time_us) (tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_wall_time_us, tablet-download-pool-9fcd6776aebf47c99f05850de83b36e6.run_wall_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.queue_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.queue_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_cpu_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_cpu_time_us) (tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_wall_time_us, tablet-download-pool-a170766f497f491ba9b45f92f2a0e751.run_wall_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.queue_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.queue_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_cpu_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_cpu_time_us) (tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_wall_time_us, tablet-download-pool-dbc6ed9f4bf34b1caa462aa9d85c7a16.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
W20240501 20:02:59.296953 12973 consensus_peers.cc:485] T 6fd958b083bb430a99130a18f91acdf7 P e3c6b84864cb4358a9d7e0f2a65ba83d -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:59.302333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7fd657f1bdcd google::LogMessage::Fail() at ??:0
I20240501 20:02:59.303689 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:59.305634 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.314895 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7fd657f1fb93 google::LogMessage::SendToLog() at ??:0
I20240501 20:02:59.315313 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.315860 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.317452 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7fd657f1b7cc google::LogMessage::Flush() at ??:0
I20240501 20:02:59.325412 14397 raft_consensus.cc:998] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.168s)
I20240501 20:02:59.325733 14397 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:59.327011 12773 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:02:59.327316 14397 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7fd657f1cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 20:02:59.342257 14397 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.344336 12773 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.344604 10838 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.346423 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7fd659385044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd659369f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:02:59.353762 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.355876 11244 consensus_queue.cc:237] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.356840 12773 raft_consensus.cc:1261] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.357172 14397 consensus_queue.cc:1035] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7fd659367efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:02:59.358793 14397 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.361547 10833 catalog_manager.cc:5484] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.361986 12773 raft_consensus.cc:2932] T 318eb8fd9972435a8e51f63bac077f09 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.364113 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 318eb8fd9972435a8e51f63bac077f09 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240501 20:02:59.364991 11181 consensus_peers.cc:485] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): Couldn't send request to peer 1679f84270414f67ac67c882220e1472. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.366384 19155 ts_tablet_manager.cc:902] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Initiating tablet copy from peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 20:02:59.366978 19155 tablet_copy_client.cc:308] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.72:35171
I20240501 20:02:59.367432 13057 tablet_copy_service.cc:140] P e3c6b84864cb4358a9d7e0f2a65ba83d: Received BeginTabletCopySession request for tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer 1679f84270414f67ac67c882220e1472 ({username='slave'} at 127.30.1.68:42723)
I20240501 20:02:59.367514 13057 tablet_copy_service.cc:161] P e3c6b84864cb4358a9d7e0f2a65ba83d: Beginning new tablet copy session on tablet 19f79cd7d4ec4b4697a2637e6d14a892 from peer 1679f84270414f67ac67c882220e1472 at {username='slave'} at 127.30.1.68:42723: session id = 1679f84270414f67ac67c882220e1472-19f79cd7d4ec4b4697a2637e6d14a892
I20240501 20:02:59.368801 13057 tablet_copy_source_session.cc:205] T 19f79cd7d4ec4b4697a2637e6d14a892 P e3c6b84864cb4358a9d7e0f2a65ba83d: Tablet Copy: opened 5 blocks and 1 log segments
@ 0x7fd65bf1a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:02:59.373364 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f79cd7d4ec4b4697a2637e6d14a892. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd65bf23a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:02:59.378818 19155 tablet_copy_client.cc:783] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 5 data blocks...
@ 0x7fd65bfa476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 20:02:59.385764 19155 tablet_copy_client.cc:647] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 1 WAL segments...
@ 0x7fd65bfb84f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 20:02:59.389060 19155 tablet_copy_client.cc:523] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fd65bfa3226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 20:02:59.392176 19155 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
@ 0x7fd65bfb2c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555cbf2b075c std::function<>::operator()() at ??:0
I20240501 20:02:59.394294 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.461s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
@ 0x7fd65936ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd65936d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 20:02:59.407773 19155 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:59.408087 19155 tablet_bootstrap.cc:492] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Bootstrap complete.
I20240501 20:02:59.408507 19155 ts_tablet_manager.cc:1367] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240501 20:02:59.409564 19155 raft_consensus.cc:348] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.409679 19155 raft_consensus.cc:729] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 20:02:59.409813 19155 consensus_queue.cc:260] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.410195 19155 ts_tablet_manager.cc:1398] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:02:59.411761 13057 tablet_copy_service.cc:342] P e3c6b84864cb4358a9d7e0f2a65ba83d: Request end of tablet copy session 1679f84270414f67ac67c882220e1472-19f79cd7d4ec4b4697a2637e6d14a892 received from {username='slave'} at 127.30.1.68:42723
I20240501 20:02:59.411831 13057 tablet_copy_service.cc:433] P e3c6b84864cb4358a9d7e0f2a65ba83d: ending tablet copy session 1679f84270414f67ac67c882220e1472-19f79cd7d4ec4b4697a2637e6d14a892 on tablet 19f79cd7d4ec4b4697a2637e6d14a892 with peer 1679f84270414f67ac67c882220e1472
@ 0x7fd65936f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555cbf2b075c std::function<>::operator()() at ??:0
@ 0x7fd65935c807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd659b9a6db start_thread at ??:0
@ 0x7fd6570d271f clone at ??:0
I20240501 20:02:59.436007 13090 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-tasksjaB1H/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:57:43.471017 13095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:43.472182 13095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:43.477072 13095 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:57:44.376963 13095 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:57:44.836563 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.836709 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.836761 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.836807 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.836850 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.836915 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:44.836974 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.837035 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.837101 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.837146 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.837206 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:44.837255 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:44.837319 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:44.837373 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 19:57:45.022181 13095 rebalancer_tool.cc:1084] tablet 8e48d711c5714ee58ebbcbd094ebf3e7: '1679f84270414f67ac67c882220e1472' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 19:57:45.022289 13095 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:45.302057 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302156 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302204 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.302244 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.302309 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302354 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.302394 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.302495 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.302541 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.302611 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.302647 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.302690 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302731 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.302773 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302803 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:45.302848 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.302896 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.302947 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.302984 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:45.303017 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:45.303051 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:45.303094 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
I20240501 19:57:45.452107 13095 rebalancer_tool.cc:1074] tablet 19bfb955b9c94fdb8e0381e5b1855248: '1679f84270414f67ac67c882220e1472' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move scheduled
I20240501 19:57:45.549149 13095 rebalancer_tool.cc:1074] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '14002a0a4c674b438c78ca5db73637cc' move scheduled
I20240501 19:57:45.719358 13095 rebalancer_tool.cc:1074] tablet 19d8d7d255494a49920849d10b800fa7: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:45.765975 13095 rebalancer_tool.cc:1084] tablet 9fcd6776aebf47c99f05850de83b36e6: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '417972166f4d40ca9b4b6471a90be6af' move ignored: Illegal state: tablet 9fcd6776aebf47c99f05850de83b36e6: replica 417972166f4d40ca9b4b6471a90be6af is already present
I20240501 19:57:45.766044 13095 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:46.373700 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.373814 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.373910 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.373952 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.373984 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.374011 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.374074 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.374147 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.374177 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:46.374207 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 602ecc7b041a4114af2ce5333edad839: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.374235 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 1679f84270414f67ac67c882220e1472 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.374258 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6fd66afac434216ace3febe00939103: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.374280 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:46.374311 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3003c6376f514bbf9c339b7fdb4b5a2c: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:46.374341 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:46.374377 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b6f2f097e1e645e19a7696601cf4fb10: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server ea26a64181784350bcd3540d0633b219
I20240501 19:57:46.454965 13095 rebalancer_tool.cc:1074] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:46.581986 13095 rebalancer_tool.cc:1074] tablet 6edb640b8cc04d20b0815b1beb45d9c8: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move scheduled
I20240501 19:57:46.727850 13095 rebalancer_tool.cc:1074] tablet 2d6621e1ff294dc9a474e076873cc72c: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move scheduled
I20240501 19:57:46.961777 13095 rebalancer_tool.cc:1084] tablet b4f0699b5bdf44499c87ff1541cd2ad3: '4456bc4917c644f8b69b4f5bfb8e1166' -> '14002a0a4c674b438c78ca5db73637cc' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 19:57:46.961850 13095 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:57:47.385321 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 83036bfcd78e41b484dc84a6bb5ad722: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.385489 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6be62a82bf214958a448a800bf9690b5: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server 417972166f4d40ca9b4b6471a90be6af
W20240501 19:57:47.385564 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 08fecbecd680417eaf65c3e7ba272052: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 14002a0a4c674b438c78ca5db73637cc
W20240501 19:57:47.385604 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d01e436b10394cc1bf201b8df8009d19: could not find any suitable replica to move from server 99348cf9a56645808d00c459fb76a6f8 to server e3c6b84864cb4358a9d7e0f2a65ba83d
W20240501 19:57:47.385650 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b7b68f5ef74432e98060f6b9cfe50ce: could not find any suitable replica to move from server 4456bc4917c644f8b69b4f5bfb8e1166 to server ea26a64181784350bcd3540d0633b219
W20240501 19:57:47.385689 13095 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 70cc64f197e4430781d4667986e6ba7b: could not find any suitable replica to move from server 92ecfb9f8f6745efa5aa2e168bafcc4e to server 417972166f4d40ca9b4b6471a90be6af
I20240501 19:57:47.494932 13095 rebalancer_tool.cc:1074] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:47.558625 13095 rebalancer_tool.cc:1074] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:47.572189 13095 rebalancer_tool.cc:1074] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:47.602999 13095 rebalancer_tool.cc:1074] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:48.830677 13095 rebalancer_tool.cc:1136] tablet 8ddd522180bd4e53906cd9572f82ab67: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:51.467484 13095 rebalancer_tool.cc:1136] tablet 156d2ba7ff2c403db2f63cbae4acfec1: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:52.597558 13095 rebalancer_tool.cc:1074] tablet 9fcd6776aebf47c99f05850de83b36e6: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:52.610993 13095 rebalancer_tool.cc:1074] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:53.307631 13095 rebalancer_tool.cc:1136] tablet 26b12620ad9a404d9ab70e8506ad1a39: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:53.362700 13095 rebalancer_tool.cc:1136] tablet 2bc8fbbdc94e49d2b8b639083e083df3: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:53.389714 13095 rebalancer_tool.cc:1136] tablet 25ea8d7ab1834abfa34ff2357aa554b0: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:53.840517 13095 rebalancer_tool.cc:1136] tablet c3ee13ec9bb1468f90cfc184455f9fff: '99348cf9a56645808d00c459fb76a6f8' -> '14002a0a4c674b438c78ca5db73637cc' move completed: OK
I20240501 19:57:53.847826 13095 rebalancer_tool.cc:1136] tablet 19d8d7d255494a49920849d10b800fa7: '4456bc4917c644f8b69b4f5bfb8e1166' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:53.854912 13095 rebalancer_tool.cc:1136] tablet 9fcd6776aebf47c99f05850de83b36e6: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:53.864595 13095 rebalancer_tool.cc:1136] tablet 2d6621e1ff294dc9a474e076873cc72c: '1679f84270414f67ac67c882220e1472' -> 'ea26a64181784350bcd3540d0633b219' move completed: OK
I20240501 19:57:53.886973 13095 rebalancer_tool.cc:1136] tablet 6edb640b8cc04d20b0815b1beb45d9c8: '99348cf9a56645808d00c459fb76a6f8' -> '417972166f4d40ca9b4b6471a90be6af' move completed: OK
I20240501 19:57:53.895596 13095 rebalancer_tool.cc:1074] tablet c49b9c5ebaab42e18fe7bbe8127d569e: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:53.921005 13095 rebalancer_tool.cc:1074] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '' move scheduled
I20240501 19:57:53.929795 13095 rebalancer_tool.cc:1074] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:54.055954 13095 rebalancer_tool.cc:1074] tablet 318eb8fd9972435a8e51f63bac077f09: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:54.090314 13095 rebalancer_tool.cc:1136] tablet 318eb8fd9972435a8e51f63bac077f09: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:54.101578 13095 rebalancer_tool.cc:1136] tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: '99348cf9a56645808d00c459fb76a6f8' -> '' move completed: OK
I20240501 19:57:55.329896 13095 rebalancer_tool.cc:1136] tablet 2def1cda07c349c69b3abe25804d3d1d: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:55.349992 13095 rebalancer_tool.cc:1136] tablet 19bfb955b9c94fdb8e0381e5b1855248: '1679f84270414f67ac67c882220e1472' -> 'e3c6b84864cb4358a9d7e0f2a65ba83d' move completed: OK
I20240501 19:57:55.369135 13095 rebalancer_tool.cc:1074] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:55.379887 13095 rebalancer_tool.cc:1074] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:55.389251 13095 rebalancer_tool.cc:1074] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move scheduled
I20240501 19:57:55.801615 13095 rebalancer_tool.cc:1136] tablet 219742bd987e4a1990233fa8c972c232: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:55.884238 13095 rebalancer_tool.cc:1136] tablet 269047cc3a504d7488747166efce71fc: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 19:57:55.918030 13095 rebalancer_tool.cc:1136] tablet c49b9c5ebaab42e18fe7bbe8127d569e: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:56.475942 13095 rebalancer_tool.cc:1074] tablet ee298406a4824f1aa2fa742b7dec1026: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:56.486644 13095 rebalancer_tool.cc:1074] tablet 6fd958b083bb430a99130a18f91acdf7: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move scheduled
I20240501 19:57:56.498746 13095 rebalancer_tool.cc:1074] tablet a170766f497f491ba9b45f92f2a0e751: '1679f84270414f67ac67c882220e1472' -> '' move scheduled
I20240501 19:57:58.932857 13095 rebalancer_tool.cc:1136] tablet 0a1bbe74dbc74fa687b0a64d3aa203f3: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:58.942555 13095 rebalancer_tool.cc:1136] tablet 19f79cd7d4ec4b4697a2637e6d14a892: '92ecfb9f8f6745efa5aa2e168bafcc4e' -> '' move completed: OK
I20240501 19:57:59.312809 13095 rebalancer_tool.cc:1136] tablet 6fd958b083bb430a99130a18f91acdf7: '4456bc4917c644f8b69b4f5bfb8e1166' -> '' move completed: OK
I20240501 19:57:59.816411 13095 rebalancer_tool.cc:1136] tablet a170766f497f491ba9b45f92f2a0e751: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
I20240501 20:02:59.172262 13095 rebalancer_tool.cc:1136] tablet ee298406a4824f1aa2fa742b7dec1026: '1679f84270414f67ac67c882220e1472' -> '' move completed: OK
Illegal state: tablet server 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): unacceptable health status UNAVAILABLE
W20240501 20:02:59.446859 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 318eb8fd9972435a8e51f63bac077f09: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:59.454646 19197 ts_tablet_manager.cc:902] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:02:59.455227 19197 tablet_copy_client.cc:308] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 20:02:59.455843 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet ee298406a4824f1aa2fa742b7dec1026 from peer 4456bc4917c644f8b69b4f5bfb8e1166 ({username='slave'} at 127.30.1.67:46073)
I20240501 20:02:59.455932 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet ee298406a4824f1aa2fa742b7dec1026 from peer 4456bc4917c644f8b69b4f5bfb8e1166 at {username='slave'} at 127.30.1.67:46073: session id = 4456bc4917c644f8b69b4f5bfb8e1166-ee298406a4824f1aa2fa742b7dec1026
I20240501 20:02:59.457129 12793 tablet_copy_source_session.cc:205] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:02:59.458117 19197 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:02:59.459986 19197 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 20:02:59.460098 19197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:59.460886 19197 tablet_copy_client.cc:783] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 5 data blocks...
I20240501 20:02:59.478158 19197 tablet_copy_client.cc:647] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:59.479907 19197 tablet_copy_client.cc:523] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:59.481474 19197 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap starting.
I20240501 20:02:59.502959 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.505312 19197 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:59.505932 19197 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Bootstrap complete.
I20240501 20:02:59.505947 11094 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:02:59.506652 19197 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20240501 20:02:59.508296 19197 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:02:59.508764 19197 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Initialized, Role: FOLLOWER
I20240501 20:02:59.508972 19197 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:02:59.509399 19197 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 20:02:59.511210 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-ee298406a4824f1aa2fa742b7dec1026 received from {username='slave'} at 127.30.1.67:46073
I20240501 20:02:59.511277 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 4456bc4917c644f8b69b4f5bfb8e1166-ee298406a4824f1aa2fa742b7dec1026 on tablet ee298406a4824f1aa2fa742b7dec1026 with peer 4456bc4917c644f8b69b4f5bfb8e1166
I20240501 20:02:59.512418 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.521493 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.524338 11354 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:02:59.531701 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892): 4921 bytes on disk
I20240501 20:02:59.532141 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(19f79cd7d4ec4b4697a2637e6d14a892) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240501 20:02:59.621838 10918 connection.cc:535] client connection to 127.30.1.72:35171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:02:59.621907 11311 connection.cc:535] client connection to 127.30.1.72:35171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:02:59.622645 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:59.622694 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:02:59.622715 11311 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:59.622747 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.644279 18803 raft_consensus.cc:979] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472: : Instructing follower 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 20:02:59.644367 18803 raft_consensus.cc:1067] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Signalling peer 92ecfb9f8f6745efa5aa2e168bafcc4e to start an election
I20240501 20:02:59.644779 11114 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7"
dest_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e"
from {username='slave'} at 127.30.1.68:41907
I20240501 20:02:59.644901 11114 raft_consensus.cc:482] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:02:59.644989 11114 raft_consensus.cc:3037] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:02:59.646011 11114 raft_consensus.cc:504] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.646328 11114 leader_election.cc:288] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 2 election: Requested vote from peers 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 20:02:59.648463 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d8d7d255494a49920849d10b800fa7" candidate_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 20:02:59.648581 11376 raft_consensus.cc:3032] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:02:59.648614 11376 raft_consensus.cc:729] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Running, Role: LEADER
I20240501 20:02:59.648732 11376 consensus_queue.cc:260] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.648910 11376 raft_consensus.cc:3037] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:02:59.649338 11049 leader_election.cc:334] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:02:59.649808 11376 raft_consensus.cc:2454] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92ecfb9f8f6745efa5aa2e168bafcc4e in term 2.
I20240501 20:02:59.650151 11050 leader_election.cc:302] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:02:59.650302 17286 raft_consensus.cc:2781] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:02:59.650518 17286 raft_consensus.cc:686] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 LEADER]: Becoming Leader. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Running, Role: LEADER
I20240501 20:02:59.650662 17286 consensus_queue.cc:237] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.651824 11292 maintenance_manager.cc:419] P 4456bc4917c644f8b69b4f5bfb8e1166: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 20:02:59.652263 11182 maintenance_manager.cc:643] P 4456bc4917c644f8b69b4f5bfb8e1166: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240501 20:02:59.653126 10832 catalog_manager.cc:5484] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: term changed from 1 to 2, leader changed from 1679f84270414f67ac67c882220e1472 (127.30.1.68) to 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66). New cstate: current_term: 2 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 20:02:59.659686 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:59.662559 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:02:59.689862 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.691666 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.691704 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.699411 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.704731 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.707320 18859 raft_consensus.cc:998] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.337s)
I20240501 20:02:59.707585 18859 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:59.708475 12773 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:02:59.708768 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.709650 18852 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:02:59.711328 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.777s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:02:59.714665 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:02:59.715970 18859 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.717552 12773 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.719934 10832 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.722879 10983 consensus_queue.cc:237] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.724193 12773 raft_consensus.cc:1261] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.724547 18859 consensus_queue.cc:1035] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:02:59.727561 10919 consensus_peers.cc:485] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371): Couldn't send request to peer 1679f84270414f67ac67c882220e1472. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25ea8d7ab1834abfa34ff2357aa554b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.731612 18859 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.732990 10828 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 25ea8d7ab1834abfa34ff2357aa554b0 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:59.733553 12773 raft_consensus.cc:2932] T 25ea8d7ab1834abfa34ff2357aa554b0 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.733870 10832 catalog_manager.cc:5484] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 8 to 9, VOTER 1679f84270414f67ac67c882220e1472 (127.30.1.68) added. New cstate: current_term: 1 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:02:59.734071 18803 raft_consensus.cc:998] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.390s)
I20240501 20:02:59.734390 18803 consensus_queue.cc:237] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.735421 11114 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.735428 12773 raft_consensus.cc:1261] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.735783 19245 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.736263 19245 consensus_queue.cc:1035] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.739722 19282 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.741478 11114 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.741585 12772 raft_consensus.cc:2932] T 6edb640b8cc04d20b0815b1beb45d9c8 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.743160 10839 catalog_manager.cc:5484] T 6edb640b8cc04d20b0815b1beb45d9c8 P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 8 to 9, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 20:02:59.755671 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.773259 19280 raft_consensus.cc:998] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.479s)
I20240501 20:02:59.773595 19280 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:59.774691 12772 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:02:59.775090 18803 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:02:59.777027 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.777372 19280 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.777683 12773 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.779680 10833 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.782160 11376 consensus_queue.cc:237] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:02:59.783084 12773 raft_consensus.cc:1261] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 1679f84270414f67ac67c882220e1472: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 20:02:59.783311 11311 consensus_peers.cc:485] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 -> Peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): Couldn't send request to peer 92ecfb9f8f6745efa5aa2e168bafcc4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.783432 19280 consensus_queue.cc:1035] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:02:59.784979 18803 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.785287 19155 ts_tablet_manager.cc:902] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Initiating tablet copy from peer 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555)
I20240501 20:02:59.785599 12773 raft_consensus.cc:2932] T 95b8c1a34cc94d29986289939b2159bb P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.785776 19155 tablet_copy_client.cc:308] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.67:33555
I20240501 20:02:59.786159 11265 tablet_copy_service.cc:140] P 4456bc4917c644f8b69b4f5bfb8e1166: Received BeginTabletCopySession request for tablet 318eb8fd9972435a8e51f63bac077f09 from peer 1679f84270414f67ac67c882220e1472 ({username='slave'} at 127.30.1.68:55769)
I20240501 20:02:59.786255 11265 tablet_copy_service.cc:161] P 4456bc4917c644f8b69b4f5bfb8e1166: Beginning new tablet copy session on tablet 318eb8fd9972435a8e51f63bac077f09 from peer 1679f84270414f67ac67c882220e1472 at {username='slave'} at 127.30.1.68:55769: session id = 1679f84270414f67ac67c882220e1472-318eb8fd9972435a8e51f63bac077f09
I20240501 20:02:59.787111 10828 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 95b8c1a34cc94d29986289939b2159bb with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:59.787499 10833 catalog_manager.cc:5484] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472 reported cstate change: config changed from index 8 to 9, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New cstate: current_term: 1 leader_uuid: "1679f84270414f67ac67c882220e1472" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:02:59.787633 11265 tablet_copy_source_session.cc:205] T 318eb8fd9972435a8e51f63bac077f09 P 4456bc4917c644f8b69b4f5bfb8e1166: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:02:59.790319 19155 ts_tablet_manager.cc:1875] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:02:59.792331 19155 ts_tablet_manager.cc:1888] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 20:02:59.792492 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318eb8fd9972435a8e51f63bac077f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:02:59.793512 19155 tablet_copy_client.cc:783] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 5 data blocks...
I20240501 20:02:59.801273 19155 tablet_copy_client.cc:647] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 1 WAL segments...
I20240501 20:02:59.802904 19155 tablet_copy_client.cc:523] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:02:59.804381 19155 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 20:02:59.814292 19155 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:02:59.814532 19155 tablet_bootstrap.cc:492] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Bootstrap complete.
I20240501 20:02:59.814693 19155 ts_tablet_manager.cc:1367] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240501 20:02:59.815100 19155 raft_consensus.cc:348] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.815197 19155 raft_consensus.cc:729] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 20:02:59.815320 19155 consensus_queue.cc:260] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:02:59.816527 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.817605 19155 ts_tablet_manager.cc:1398] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:02:59.818049 11265 tablet_copy_service.cc:342] P 4456bc4917c644f8b69b4f5bfb8e1166: Request end of tablet copy session 1679f84270414f67ac67c882220e1472-318eb8fd9972435a8e51f63bac077f09 received from {username='slave'} at 127.30.1.68:55769
I20240501 20:02:59.818114 11265 tablet_copy_service.cc:433] P 4456bc4917c644f8b69b4f5bfb8e1166: ending tablet copy session 1679f84270414f67ac67c882220e1472-318eb8fd9972435a8e51f63bac077f09 on tablet 318eb8fd9972435a8e51f63bac077f09 with peer 1679f84270414f67ac67c882220e1472
I20240501 20:02:59.821048 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.821132 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.829047 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:02:59.830617 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:02:59.831554 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.863348 19279 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (300.930s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:02:59.916177 18594 raft_consensus.cc:998] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.375s)
I20240501 20:02:59.916491 18594 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:02:59.917994 12773 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:02:59.918365 18594 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:02:59.920083 19389 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.921418 12773 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.923400 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:02:59.923393 10839 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 7 to 8, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
I20240501 20:02:59.927659 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.932945 11114 consensus_queue.cc:237] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:02:59.934100 12773 raft_consensus.cc:1261] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:02:59.934459 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:02:59.934530 18594 consensus_queue.cc:1035] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:02:59.936157 11050 consensus_peers.cc:485] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): Couldn't send request to peer 99348cf9a56645808d00c459fb76a6f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:02:59.937445 18594 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:59.939580 10834 catalog_manager.cc:5484] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 8 to 9, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New cstate: current_term: 1 leader_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:59.940351 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.940385 12773 raft_consensus.cc:2932] T 26b12620ad9a404d9ab70e8506ad1a39 P 14002a0a4c674b438c78ca5db73637cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:02:59.941918 10827 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 26b12620ad9a404d9ab70e8506ad1a39 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:02:59.942862 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:02:59.964170 11244 raft_consensus.cc:1203] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 20:02:59.976886 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:59.977545 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:02:59.999433 19413 raft_consensus.cc:998] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.501s)
I20240501 20:02:59.999771 19413 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:00.000998 11114 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:03:00.001374 19413 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:03:00.003165 18849 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:03:00.003566 11114 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:03:00.006291 10836 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e reported cstate change: config changed from index 10 to 11, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:03:00.010160 12773 consensus_queue.cc:237] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.011551 11114 raft_consensus.cc:1261] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 20:03:00.011667 12709 consensus_peers.cc:485] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc -> Peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): Couldn't send request to peer 99348cf9a56645808d00c459fb76a6f8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bc8fbbdc94e49d2b8b639083e083df3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:00.011967 19413 consensus_queue.cc:1035] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:03:00.013424 18849 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.013870 11112 raft_consensus.cc:2932] T 2bc8fbbdc94e49d2b8b639083e083df3 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.015761 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 2bc8fbbdc94e49d2b8b639083e083df3 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:03:00.016391 10837 catalog_manager.cc:5484] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.034163 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09): 4579 bytes on disk
I20240501 20:03:00.034595 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(318eb8fd9972435a8e51f63bac077f09) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 20:03:00.034749 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.036612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.036840 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.045663 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:03:00.047191 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.054433 11376 raft_consensus.cc:1261] T 19d8d7d255494a49920849d10b800fa7 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Refusing update from remote peer 92ecfb9f8f6745efa5aa2e168bafcc4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:03:00.054764 19389 consensus_queue.cc:1035] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:03:00.062449 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:00.134936 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (301.200s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:00.136524 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:00.142627 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.142724 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.144708 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.153620 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.155649 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:03:00.234515 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.235175 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.240768 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:03:00.247978 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.247978 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.286298 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:03:00.287282 11244 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:03:00.287671 19419 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:03:00.289144 18849 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:00.289510 11244 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:00.291271 10834 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 10 to 11, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:00.294071 10961 tablet_service.cc:1514] Processing DeleteTablet for tablet ee298406a4824f1aa2fa742b7dec1026 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41432
I20240501 20:03:00.294407 19545 tablet_replica.cc:330] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 20:03:00.294543 19545 raft_consensus.cc:2229] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:03:00.294708 19545 raft_consensus.cc:2258] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:03:00.294674 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.295312 19545 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:03:00.295686 11244 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 20:03:00.295763 12709 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): Couldn't send request to peer 99348cf9a56645808d00c459fb76a6f8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:00.296018 18849 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:03:00.297292 19419 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.297621 11244 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.297977 19545 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:03:00.298034 19545 log.cc:1196] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee298406a4824f1aa2fa742b7dec1026
I20240501 20:03:00.298578 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet ee298406a4824f1aa2fa742b7dec1026 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 20:03:00.299134 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ee298406a4824f1aa2fa742b7dec1026 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:03:00.299655 10834 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 11 to 12, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.300689 19155 ts_tablet_manager.cc:902] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Initiating tablet copy from peer 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 20:03:00.301105 19155 tablet_copy_client.cc:308] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.65:40027
I20240501 20:03:00.304081 11002 tablet_copy_service.cc:140] P 99348cf9a56645808d00c459fb76a6f8: Received BeginTabletCopySession request for tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 1679f84270414f67ac67c882220e1472 ({username='slave'} at 127.30.1.68:45501)
I20240501 20:03:00.304172 11002 tablet_copy_service.cc:161] P 99348cf9a56645808d00c459fb76a6f8: Beginning new tablet copy session on tablet 25ea8d7ab1834abfa34ff2357aa554b0 from peer 1679f84270414f67ac67c882220e1472 at {username='slave'} at 127.30.1.68:45501: session id = 1679f84270414f67ac67c882220e1472-25ea8d7ab1834abfa34ff2357aa554b0
I20240501 20:03:00.305123 11002 tablet_copy_source_session.cc:205] T 25ea8d7ab1834abfa34ff2357aa554b0 P 99348cf9a56645808d00c459fb76a6f8: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:00.305969 19155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ea8d7ab1834abfa34ff2357aa554b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:00.307698 19155 tablet_copy_client.cc:783] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:00.311488 19155 tablet_copy_client.cc:647] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:00.312656 19155 tablet_copy_client.cc:523] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:00.312809 19554 ts_tablet_manager.cc:902] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Initiating tablet copy from peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739)
I20240501 20:03:00.313336 19554 tablet_copy_client.cc:308] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.66:38739
I20240501 20:03:00.313740 11134 tablet_copy_service.cc:140] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Received BeginTabletCopySession request for tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 99348cf9a56645808d00c459fb76a6f8 ({username='slave'} at 127.30.1.65:58859)
I20240501 20:03:00.313835 11134 tablet_copy_service.cc:161] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Beginning new tablet copy session on tablet 26b12620ad9a404d9ab70e8506ad1a39 from peer 99348cf9a56645808d00c459fb76a6f8 at {username='slave'} at 127.30.1.65:58859: session id = 99348cf9a56645808d00c459fb76a6f8-26b12620ad9a404d9ab70e8506ad1a39
I20240501 20:03:00.313920 19155 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Bootstrap starting.
I20240501 20:03:00.315074 11134 tablet_copy_source_session.cc:205] T 26b12620ad9a404d9ab70e8506ad1a39 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:00.315809 19554 ts_tablet_manager.cc:1875] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:03:00.317301 19554 ts_tablet_manager.cc:1888] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 20:03:00.317431 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b12620ad9a404d9ab70e8506ad1a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:00.318166 19554 tablet_copy_client.cc:783] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:00.322247 19155 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:00.322289 19554 tablet_copy_client.cc:647] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:00.322480 19155 tablet_bootstrap.cc:492] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Bootstrap complete.
I20240501 20:03:00.322619 19155 ts_tablet_manager.cc:1367] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240501 20:03:00.323006 19155 raft_consensus.cc:348] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.323114 19155 raft_consensus.cc:729] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1679f84270414f67ac67c882220e1472, State: Initialized, Role: FOLLOWER
I20240501 20:03:00.323267 19155 consensus_queue.cc:260] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.323542 19155 ts_tablet_manager.cc:1398] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:03:00.323539 19554 tablet_copy_client.cc:523] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:00.323961 11002 tablet_copy_service.cc:342] P 99348cf9a56645808d00c459fb76a6f8: Request end of tablet copy session 1679f84270414f67ac67c882220e1472-25ea8d7ab1834abfa34ff2357aa554b0 received from {username='slave'} at 127.30.1.68:45501
I20240501 20:03:00.324026 11002 tablet_copy_service.cc:433] P 99348cf9a56645808d00c459fb76a6f8: ending tablet copy session 1679f84270414f67ac67c882220e1472-25ea8d7ab1834abfa34ff2357aa554b0 on tablet 25ea8d7ab1834abfa34ff2357aa554b0 with peer 1679f84270414f67ac67c882220e1472
I20240501 20:03:00.324699 19554 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 20:03:00.331421 19554 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:00.331614 19554 tablet_bootstrap.cc:492] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap complete.
I20240501 20:03:00.331738 19554 ts_tablet_manager.cc:1367] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:03:00.332048 19554 raft_consensus.cc:348] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.332129 19554 raft_consensus.cc:729] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 20:03:00.332224 19554 consensus_queue.cc:260] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.332464 19554 ts_tablet_manager.cc:1398] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:03:00.333115 11134 tablet_copy_service.cc:342] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Request end of tablet copy session 99348cf9a56645808d00c459fb76a6f8-26b12620ad9a404d9ab70e8506ad1a39 received from {username='slave'} at 127.30.1.65:58859
I20240501 20:03:00.333189 11134 tablet_copy_service.cc:433] P 92ecfb9f8f6745efa5aa2e168bafcc4e: ending tablet copy session 99348cf9a56645808d00c459fb76a6f8-26b12620ad9a404d9ab70e8506ad1a39 on tablet 26b12620ad9a404d9ab70e8506ad1a39 with peer 99348cf9a56645808d00c459fb76a6f8
I20240501 20:03:00.344197 11376 raft_consensus.cc:1203] T 318eb8fd9972435a8e51f63bac077f09 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:03:00.357674 19561 ts_tablet_manager.cc:902] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Initiating tablet copy from peer 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 20:03:00.358079 19561 tablet_copy_client.cc:308] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.68:34371
I20240501 20:03:00.358467 11395 tablet_copy_service.cc:140] P 1679f84270414f67ac67c882220e1472: Received BeginTabletCopySession request for tablet 95b8c1a34cc94d29986289939b2159bb from peer 92ecfb9f8f6745efa5aa2e168bafcc4e ({username='slave'} at 127.30.1.66:56439)
I20240501 20:03:00.358559 11395 tablet_copy_service.cc:161] P 1679f84270414f67ac67c882220e1472: Beginning new tablet copy session on tablet 95b8c1a34cc94d29986289939b2159bb from peer 92ecfb9f8f6745efa5aa2e168bafcc4e at {username='slave'} at 127.30.1.66:56439: session id = 92ecfb9f8f6745efa5aa2e168bafcc4e-95b8c1a34cc94d29986289939b2159bb
I20240501 20:03:00.359349 11395 tablet_copy_source_session.cc:205] T 95b8c1a34cc94d29986289939b2159bb P 1679f84270414f67ac67c882220e1472: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:00.360023 19561 ts_tablet_manager.cc:1875] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:03:00.361804 19561 ts_tablet_manager.cc:1888] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:03:00.361946 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8c1a34cc94d29986289939b2159bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:00.362708 19561 tablet_copy_client.cc:783] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:00.366688 19561 tablet_copy_client.cc:647] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:00.367842 19561 tablet_copy_client.cc:523] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:00.368825 19561 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 20:03:00.375293 19561 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:00.375520 19561 tablet_bootstrap.cc:492] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap complete.
I20240501 20:03:00.375670 19561 ts_tablet_manager.cc:1367] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:03:00.375982 19561 raft_consensus.cc:348] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:00.376063 19561 raft_consensus.cc:729] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 20:03:00.376173 19561 consensus_queue.cc:260] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:00.376441 19561 ts_tablet_manager.cc:1398] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:03:00.376848 11395 tablet_copy_service.cc:342] P 1679f84270414f67ac67c882220e1472: Request end of tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-95b8c1a34cc94d29986289939b2159bb received from {username='slave'} at 127.30.1.66:56439
I20240501 20:03:00.376933 11395 tablet_copy_service.cc:433] P 1679f84270414f67ac67c882220e1472: ending tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-95b8c1a34cc94d29986289939b2159bb on tablet 95b8c1a34cc94d29986289939b2159bb with peer 92ecfb9f8f6745efa5aa2e168bafcc4e
I20240501 20:03:00.413497 19554 ts_tablet_manager.cc:902] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:03:00.413897 19554 tablet_copy_client.cc:308] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 20:03:00.414337 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 99348cf9a56645808d00c459fb76a6f8 ({username='slave'} at 127.30.1.65:46897)
I20240501 20:03:00.414417 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet 2bc8fbbdc94e49d2b8b639083e083df3 from peer 99348cf9a56645808d00c459fb76a6f8 at {username='slave'} at 127.30.1.65:46897: session id = 99348cf9a56645808d00c459fb76a6f8-2bc8fbbdc94e49d2b8b639083e083df3
I20240501 20:03:00.415130 12793 tablet_copy_source_session.cc:205] T 2bc8fbbdc94e49d2b8b639083e083df3 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:00.415829 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc8fbbdc94e49d2b8b639083e083df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:00.417405 19554 tablet_copy_client.cc:783] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:00.421283 19554 tablet_copy_client.cc:647] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:00.422358 19554 tablet_copy_client.cc:523] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:00.423285 19554 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 20:03:00.430647 19554 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:00.430873 19554 tablet_bootstrap.cc:492] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap complete.
I20240501 20:03:00.431023 19554 ts_tablet_manager.cc:1367] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:03:00.431317 19554 raft_consensus.cc:348] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.431399 19554 raft_consensus.cc:729] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 20:03:00.431488 19554 consensus_queue.cc:260] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.431725 19554 ts_tablet_manager.cc:1398] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:03:00.432174 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 99348cf9a56645808d00c459fb76a6f8-2bc8fbbdc94e49d2b8b639083e083df3 received from {username='slave'} at 127.30.1.65:46897
I20240501 20:03:00.432235 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 99348cf9a56645808d00c459fb76a6f8-2bc8fbbdc94e49d2b8b639083e083df3 on tablet 2bc8fbbdc94e49d2b8b639083e083df3 with peer 99348cf9a56645808d00c459fb76a6f8
W20240501 20:03:00.436414 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.461337 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3): 4921 bytes on disk
I20240501 20:03:00.461706 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(2bc8fbbdc94e49d2b8b639083e083df3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 20:03:00.462255 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39): 4580 bytes on disk
I20240501 20:03:00.462527 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(26b12620ad9a404d9ab70e8506ad1a39) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
W20240501 20:03:00.480006 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 318eb8fd9972435a8e51f63bac077f09: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:00.481047 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (301.548s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:00.481788 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:03:00.484622 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 20:03:00.536690 11423 maintenance_manager.cc:419] P 1679f84270414f67ac67c882220e1472: Scheduling UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0): 4921 bytes on disk
I20240501 20:03:00.537261 11313 maintenance_manager.cc:643] P 1679f84270414f67ac67c882220e1472: UndoDeltaBlockGCOp(25ea8d7ab1834abfa34ff2357aa554b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 20:03:00.538923 19413 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.539022 19413 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:03:00.539433 19413 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 20:03:00.539902 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "14002a0a4c674b438c78ca5db73637cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
W20240501 20:03:00.541694 12708 leader_election.cc:334] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.541765 12708 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14002a0a4c674b438c78ca5db73637cc; no voters: 99348cf9a56645808d00c459fb76a6f8, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:00.541887 19413 raft_consensus.cc:2726] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:00.551107 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb): 4581 bytes on disk
I20240501 20:03:00.551568 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(95b8c1a34cc94d29986289939b2159bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 20:03:00.557408 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.557898 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (301.623s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:00.558307 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:03:00.558945 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.559659 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:00.559855 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:00.569017 19280 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:03:00.569105 19280 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.569456 19280 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 20:03:00.570039 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "1679f84270414f67ac67c882220e1472" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
I20240501 20:03:00.570767 10961 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:00.577383 11311 leader_election.cc:334] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.577452 11311 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1679f84270414f67ac67c882220e1472; no voters: 4456bc4917c644f8b69b4f5bfb8e1166, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:00.577637 19280 raft_consensus.cc:2726] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:00.592413 18852 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.592528 18852 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:03:00.592898 18852 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:03:00.593269 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "14002a0a4c674b438c78ca5db73637cc" is_pre_election: true
I20240501 20:03:00.593492 12773 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 2.
I20240501 20:03:00.593797 10917 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14002a0a4c674b438c78ca5db73637cc, 99348cf9a56645808d00c459fb76a6f8; no voters:
W20240501 20:03:00.593901 10918 leader_election.cc:334] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.593972 18852 raft_consensus.cc:2781] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:03:00.594030 18852 raft_consensus.cc:482] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.594094 18852 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:03:00.594982 18852 raft_consensus.cc:504] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:03:00.595335 18852 leader_election.cc:288] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 election: Requested vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:03:00.595741 12773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2def1cda07c349c69b3abe25804d3d1d" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "14002a0a4c674b438c78ca5db73637cc"
I20240501 20:03:00.595889 12773 raft_consensus.cc:3037] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:03:00.596266 10918 leader_election.cc:334] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.596812 12773 raft_consensus.cc:2454] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99348cf9a56645808d00c459fb76a6f8 in term 3.
I20240501 20:03:00.597079 10917 leader_election.cc:302] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14002a0a4c674b438c78ca5db73637cc, 99348cf9a56645808d00c459fb76a6f8; no voters: e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:00.597267 18852 raft_consensus.cc:2781] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:03:00.597357 18852 raft_consensus.cc:686] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [term 3 LEADER]: Becoming Leader. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Running, Role: LEADER
I20240501 20:03:00.597514 18852 consensus_queue.cc:237] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } }
I20240501 20:03:00.599649 10837 catalog_manager.cc:5484] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: term changed from 2 to 3, leader changed from e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) to 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65). New cstate: current_term: 3 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } }
W20240501 20:03:00.648016 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:03:00.657058 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.659718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.661350 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:00.667599 19279 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.667696 19279 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.668058 19279 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67:33555), e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171)
I20240501 20:03:00.668844 11244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "99348cf9a56645808d00c459fb76a6f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" is_pre_election: true
W20240501 20:03:00.669332 10918 leader_election.cc:334] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.669394 10918 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 4456bc4917c644f8b69b4f5bfb8e1166, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:00.669577 19279 raft_consensus.cc:2726] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:00.675422 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:00.678894 10919 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:03:00.682773 11376 raft_consensus.cc:1203] T 25ea8d7ab1834abfa34ff2357aa554b0 P 1679f84270414f67ac67c882220e1472 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:03:00.686630 11224 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:03:00.720531 10982 raft_consensus.cc:1203] T 26b12620ad9a404d9ab70e8506ad1a39 P 99348cf9a56645808d00c459fb76a6f8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:03:00.761066 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:00.763267 19156 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.763366 19156 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.763769 19156 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 20:03:00.764221 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472" is_pre_election: true
I20240501 20:03:00.764344 11376 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 2.
I20240501 20:03:00.764614 11181 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 20:03:00.765257 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:00.765815 11180 leader_election.cc:334] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.766153 19156 raft_consensus.cc:2781] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:03:00.766206 19156 raft_consensus.cc:482] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.766245 19156 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:03:00.767325 19156 raft_consensus.cc:504] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.767617 19156 leader_election.cc:288] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: Requested vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 1679f84270414f67ac67c882220e1472 (127.30.1.68:34371)
I20240501 20:03:00.768085 11376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f79cd7d4ec4b4697a2637e6d14a892" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "1679f84270414f67ac67c882220e1472"
I20240501 20:03:00.768190 11376 raft_consensus.cc:3037] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:03:00.768221 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:00.769064 11376 raft_consensus.cc:2454] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 3.
I20240501 20:03:00.769315 11181 leader_election.cc:302] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1679f84270414f67ac67c882220e1472, 4456bc4917c644f8b69b4f5bfb8e1166; no voters:
I20240501 20:03:00.769922 19156 raft_consensus.cc:2781] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:03:00.770088 19156 raft_consensus.cc:686] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 20:03:00.770206 19156 consensus_queue.cc:237] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } }
I20240501 20:03:00.771865 10834 catalog_manager.cc:5484] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 2 to 3, leader changed from e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 3 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 } } }
I20240501 20:03:00.775420 11224 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:03:00.781956 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:00.785193 11180 leader_election.cc:334] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.787375 19701 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.787498 19701 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.788774 19701 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 20:03:00.788833 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8" is_pre_election: true
I20240501 20:03:00.788996 10982 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 2.
I20240501 20:03:00.789265 11181 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
W20240501 20:03:00.789361 11180 leader_election.cc:334] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.789583 19701 raft_consensus.cc:2781] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:03:00.789665 19701 raft_consensus.cc:482] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.789741 19701 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:03:00.791500 19701 raft_consensus.cc:504] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.792429 19701 leader_election.cc:288] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: Requested vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027)
I20240501 20:03:00.792482 10982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d2ba7ff2c403db2f63cbae4acfec1" candidate_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "99348cf9a56645808d00c459fb76a6f8"
I20240501 20:03:00.792601 10982 raft_consensus.cc:3037] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:03:00.793521 10982 raft_consensus.cc:2454] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4456bc4917c644f8b69b4f5bfb8e1166 in term 3.
I20240501 20:03:00.793788 11181 leader_election.cc:302] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456bc4917c644f8b69b4f5bfb8e1166, 99348cf9a56645808d00c459fb76a6f8; no voters:
W20240501 20:03:00.794111 11180 leader_election.cc:334] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.794271 19701 raft_consensus.cc:2781] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:03:00.794564 19701 raft_consensus.cc:686] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 3 LEADER]: Becoming Leader. State: Replica: 4456bc4917c644f8b69b4f5bfb8e1166, State: Running, Role: LEADER
I20240501 20:03:00.794706 19701 consensus_queue.cc:237] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.796248 10834 catalog_manager.cc:5484] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 reported cstate change: term changed from 2 to 3, leader changed from e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72) to 4456bc4917c644f8b69b4f5bfb8e1166 (127.30.1.67). New cstate: current_term: 3 leader_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:00.847836 19554 ts_tablet_manager.cc:902] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:03:00.848256 19554 tablet_copy_client.cc:308] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 20:03:00.848634 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet ee298406a4824f1aa2fa742b7dec1026 from peer 99348cf9a56645808d00c459fb76a6f8 ({username='slave'} at 127.30.1.65:46897)
I20240501 20:03:00.848735 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet ee298406a4824f1aa2fa742b7dec1026 from peer 99348cf9a56645808d00c459fb76a6f8 at {username='slave'} at 127.30.1.65:46897: session id = 99348cf9a56645808d00c459fb76a6f8-ee298406a4824f1aa2fa742b7dec1026
I20240501 20:03:00.849830 12793 tablet_copy_source_session.cc:205] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:00.851061 19554 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:03:00.853228 19554 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.10
I20240501 20:03:00.853391 19554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:00.854308 19554 tablet_copy_client.cc:783] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:00.859827 19554 tablet_copy_client.cc:647] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:00.860996 19554 tablet_copy_client.cc:523] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:00.862040 19554 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap starting.
I20240501 20:03:00.870465 19554 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:00.870683 19554 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Bootstrap complete.
I20240501 20:03:00.870829 19554 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 20:03:00.871155 19554 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.871251 19554 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99348cf9a56645808d00c459fb76a6f8, State: Initialized, Role: FOLLOWER
I20240501 20:03:00.871374 19554 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:00.871618 19554 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:03:00.872052 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 99348cf9a56645808d00c459fb76a6f8-ee298406a4824f1aa2fa742b7dec1026 received from {username='slave'} at 127.30.1.65:46897
I20240501 20:03:00.872128 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 99348cf9a56645808d00c459fb76a6f8-ee298406a4824f1aa2fa742b7dec1026 on tablet ee298406a4824f1aa2fa742b7dec1026 with peer 99348cf9a56645808d00c459fb76a6f8
I20240501 20:03:00.878695 18852 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:00.878782 18852 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:00.879104 18852 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:00.880241 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:00.882871 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:00.882926 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:00.882999 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:00.883392 18852 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:00.888972 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:00.890692 11112 raft_consensus.cc:1203] T 95b8c1a34cc94d29986289939b2159bb P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 20:03:00.891939 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.894707 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.894711 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:00.964332 11030 maintenance_manager.cc:419] P 99348cf9a56645808d00c459fb76a6f8: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 20:03:00.964720 10920 maintenance_manager.cc:643] P 99348cf9a56645808d00c459fb76a6f8: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 20:03:00.966125 19279 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (302.031s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:00.967101 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:00.969635 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (302.037s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:01.018448 12773 raft_consensus.cc:1261] T 2def1cda07c349c69b3abe25804d3d1d P 14002a0a4c674b438c78ca5db73637cc [term 3 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:03:01.018824 18852 consensus_queue.cc:1035] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:03:01.021607 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:03:01.031950 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:01.032114 10982 raft_consensus.cc:1203] T 2bc8fbbdc94e49d2b8b639083e083df3 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 20:03:01.176060 18852 raft_consensus.cc:998] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8: Attempting to remove follower 417972166f4d40ca9b4b6471a90be6af from the Raft config. Reason: Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (300.317s)
I20240501 20:03:01.176440 18852 consensus_queue.cc:237] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 11, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } }
I20240501 20:03:01.177950 11112 raft_consensus.cc:1261] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 5 FOLLOWER]: Refusing update from remote peer 99348cf9a56645808d00c459fb76a6f8: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
W20240501 20:03:01.179880 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:01.179916 19279 consensus_queue.cc:1035] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:03:01.184077 19279 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 [term 5 LEADER]: Committing config change with OpId 5.13: config changed from index 11 to 13, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:01.186419 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.186779 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.186905 10839 catalog_manager.cc:5484] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 reported cstate change: config changed from index 11 to 13, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New cstate: current_term: 5 leader_uuid: "99348cf9a56645808d00c459fb76a6f8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
I20240501 20:03:01.187139 11112 raft_consensus.cc:2932] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 5 FOLLOWER]: Committing config change with OpId 5.13: config changed from index 11 to 13, VOTER 417972166f4d40ca9b4b6471a90be6af (127.30.1.69) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } }
W20240501 20:03:01.189582 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:01.190078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.191711 10830 maintenance_manager.cc:419] P 06cf8280f20843729e99a32633f62d20: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7603 bytes on disk
I20240501 20:03:01.193985 10829 maintenance_manager.cc:643] P 06cf8280f20843729e99a32633f62d20: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":159,"thread_start_us":67,"threads_started":1}
W20240501 20:03:01.200565 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:01.202244 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.210076 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:01.256243 11376 raft_consensus.cc:1261] T 19f79cd7d4ec4b4697a2637e6d14a892 P 1679f84270414f67ac67c882220e1472 [term 3 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:03:01.256654 19701 consensus_queue.cc:1035] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1679f84270414f67ac67c882220e1472" member_type: VOTER last_known_addr { host: "127.30.1.68" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:03:01.260392 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:01.300566 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.302052 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.304430 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.305439 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:01.313040 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.321393 10982 raft_consensus.cc:1261] T 156d2ba7ff2c403db2f63cbae4acfec1 P 99348cf9a56645808d00c459fb76a6f8 [term 3 FOLLOWER]: Refusing update from remote peer 4456bc4917c644f8b69b4f5bfb8e1166: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:03:01.321815 19156 consensus_queue.cc:1035] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:03:01.330772 10982 raft_consensus.cc:1203] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 20:03:01.384267 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (302.451s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:01.395993 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.398954 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.400715 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.402936 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.403317 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:01.426479 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:01.482522 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (302.548s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:01.483543 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:01.490077 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.491017 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.492275 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.492535 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:01.505112 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:01.509441 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:01.538892 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } }
I20240501 20:03:01.539839 11244 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:03:01.540175 19413 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:03:01.541736 19951 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:01.542085 11244 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:01.544031 10837 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 12 to 13, VOTER 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65) evicted. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } }
I20240501 20:03:01.546761 10961 tablet_service.cc:1514] Processing DeleteTablet for tablet ee298406a4824f1aa2fa742b7dec1026 with delete_type TABLET_DATA_TOMBSTONED (TS 99348cf9a56645808d00c459fb76a6f8 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:41432
I20240501 20:03:01.547070 19952 tablet_replica.cc:330] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: stopping tablet replica
I20240501 20:03:01.547300 19952 raft_consensus.cc:2229] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:03:01.547422 19952 raft_consensus.cc:2258] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:03:01.547488 12773 consensus_queue.cc:237] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:01.547905 19952 ts_tablet_manager.cc:1875] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:03:01.548529 11244 raft_consensus.cc:1261] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Refusing update from remote peer 14002a0a4c674b438c78ca5db73637cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20240501 20:03:01.548741 12708 consensus_peers.cc:485] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc -> Peer 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66:38739): Couldn't send request to peer 92ecfb9f8f6745efa5aa2e168bafcc4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee298406a4824f1aa2fa742b7dec1026. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:03:01.548933 19951 consensus_queue.cc:1035] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:03:01.550266 19413 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:03:01.550585 19952 ts_tablet_manager.cc:1888] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 20:03:01.550655 19952 log.cc:1196] T ee298406a4824f1aa2fa742b7dec1026 P 99348cf9a56645808d00c459fb76a6f8: Deleting WAL directory at /tmp/dist-test-tasksjaB1H/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714593357797533-30725-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee298406a4824f1aa2fa742b7dec1026
I20240501 20:03:01.551144 10828 catalog_manager.cc:4830] TS 99348cf9a56645808d00c459fb76a6f8 (127.30.1.65:40027): tablet ee298406a4824f1aa2fa742b7dec1026 (table rebalance_test_table_7 [id=3003c6376f514bbf9c339b7fdb4b5a2c]) successfully deleted
I20240501 20:03:01.551741 10826 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ee298406a4824f1aa2fa742b7dec1026 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240501 20:03:01.551949 11244 raft_consensus.cc:2932] T ee298406a4824f1aa2fa742b7dec1026 P 4456bc4917c644f8b69b4f5bfb8e1166 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
I20240501 20:03:01.552565 10837 catalog_manager.cc:5484] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc reported cstate change: config changed from index 13 to 14, VOTER 92ecfb9f8f6745efa5aa2e168bafcc4e (127.30.1.66) added. New cstate: current_term: 2 leader_uuid: "14002a0a4c674b438c78ca5db73637cc" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } }
W20240501 20:03:01.629365 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:01.805464 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.807731 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.807909 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:01.813325 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.814122 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.906849 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:01.908063 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.913439 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (302.979s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:01.913885 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:01.914443 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:01.917999 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:01.918393 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:01.922865 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:01.991339 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (303.058s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:02.002857 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.005192 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.005461 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.007634 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.008570 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.011297 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.074525 20107 ts_tablet_manager.cc:902] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Initiating tablet copy from peer 14002a0a4c674b438c78ca5db73637cc (127.30.1.70:35165)
I20240501 20:03:02.075080 20107 tablet_copy_client.cc:308] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Beginning tablet copy session from remote peer at address 127.30.1.70:35165
I20240501 20:03:02.075774 12793 tablet_copy_service.cc:140] P 14002a0a4c674b438c78ca5db73637cc: Received BeginTabletCopySession request for tablet ee298406a4824f1aa2fa742b7dec1026 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e ({username='slave'} at 127.30.1.66:41419)
I20240501 20:03:02.075870 12793 tablet_copy_service.cc:161] P 14002a0a4c674b438c78ca5db73637cc: Beginning new tablet copy session on tablet ee298406a4824f1aa2fa742b7dec1026 from peer 92ecfb9f8f6745efa5aa2e168bafcc4e at {username='slave'} at 127.30.1.66:41419: session id = 92ecfb9f8f6745efa5aa2e168bafcc4e-ee298406a4824f1aa2fa742b7dec1026
I20240501 20:03:02.076849 12793 tablet_copy_source_session.cc:205] T ee298406a4824f1aa2fa742b7dec1026 P 14002a0a4c674b438c78ca5db73637cc: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:03:02.077816 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee298406a4824f1aa2fa742b7dec1026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:03:02.080464 20107 tablet_copy_client.cc:783] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 5 data blocks...
I20240501 20:03:02.090538 20107 tablet_copy_client.cc:647] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:03:02.096303 20107 tablet_copy_client.cc:523] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:03:02.097541 20107 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap starting.
I20240501 20:03:02.112600 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.113925 20107 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:03:02.114189 20107 tablet_bootstrap.cc:492] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Bootstrap complete.
I20240501 20:03:02.114310 20107 ts_tablet_manager.cc:1367] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240501 20:03:02.114711 20107 raft_consensus.cc:348] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:02.114840 20107 raft_consensus.cc:729] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ecfb9f8f6745efa5aa2e168bafcc4e, State: Initialized, Role: FOLLOWER
I20240501 20:03:02.116027 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.116060 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.116186 20107 consensus_queue.cc:260] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "14002a0a4c674b438c78ca5db73637cc" member_type: VOTER last_known_addr { host: "127.30.1.70" port: 35165 } } peers { permanent_uuid: "4456bc4917c644f8b69b4f5bfb8e1166" member_type: VOTER last_known_addr { host: "127.30.1.67" port: 33555 } } peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } }
I20240501 20:03:02.116546 20107 ts_tablet_manager.cc:1398] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:03:02.117267 12793 tablet_copy_service.cc:342] P 14002a0a4c674b438c78ca5db73637cc: Request end of tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-ee298406a4824f1aa2fa742b7dec1026 received from {username='slave'} at 127.30.1.66:41419
I20240501 20:03:02.117342 12793 tablet_copy_service.cc:433] P 14002a0a4c674b438c78ca5db73637cc: ending tablet copy session 92ecfb9f8f6745efa5aa2e168bafcc4e-ee298406a4824f1aa2fa742b7dec1026 on tablet ee298406a4824f1aa2fa742b7dec1026 with peer 92ecfb9f8f6745efa5aa2e168bafcc4e
I20240501 20:03:02.125092 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.125257 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.136886 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.189371 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.220508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.226312 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.227492 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.228067 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.237735 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.237818 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.275136 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.301671 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (303.367s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:02.302577 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:02.307132 11161 maintenance_manager.cc:419] P 92ecfb9f8f6745efa5aa2e168bafcc4e: Scheduling UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026): 4581 bytes on disk
I20240501 20:03:02.308037 11051 maintenance_manager.cc:643] P 92ecfb9f8f6745efa5aa2e168bafcc4e: UndoDeltaBlockGCOp(ee298406a4824f1aa2fa742b7dec1026) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 20:03:02.317862 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.318099 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.318850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.327236 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.330402 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.349355 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:03:02.447424 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:02.452850 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:02.473455 11112 raft_consensus.cc:1203] T ee298406a4824f1aa2fa742b7dec1026 P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
W20240501 20:03:02.485853 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 20:03:02.488076 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:03:02.543221 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (303.610s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:02.614737 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.614970 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.615497 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.616029 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.617952 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.712417 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.713045 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.714406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:02.715623 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.716795 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:02.787832 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 20:03:02.805487 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:02.808447 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:03:02.809065 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.809177 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:02.809407 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.812181 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:02.913381 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (303.979s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:02.914206 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:03.021045 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:03.046422 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (304.113s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:03.047367 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:03:03.109234 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.110882 19800 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:03.110981 19800 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:03.111366 19800 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 20:03:03.113737 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.114282 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.116333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:03.117727 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:03.117889 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:03.117936 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:03.118146 19800 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:03.121424 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:03.205087 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.205122 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:03.208314 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.210722 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:03.224494 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.303882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:03.307709 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.307719 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.315580 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.318802 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:03.330825 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:03.344379 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (304.410s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:03.345167 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:03.465124 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (304.532s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:03.499318 10828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240501 20:03:03.594393 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:03.606873 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.610394 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.610441 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:03.624282 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.626451 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:03.652595 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:03:03.689098 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:03.698653 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:03.711542 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.711580 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:03.713887 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:03.714293 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:03.722149 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:03:03.722869 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.795482 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (304.861s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:03.797535 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:03.802874 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:03.805689 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.807121 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:03.807592 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:03.808132 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:03.961660 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 20:03:03.989197 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:04.035967 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6edb640b8cc04d20b0815b1beb45d9c8: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:04.079311 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (305.146s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:04.104311 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:04.109799 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.112455 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.124501 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.133277 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:04.201570 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (305.267s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:04.202523 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:04.209127 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:04.211416 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:04.213999 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.219818 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:04.220388 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.239936 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.285590 19756 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:04.285691 19756 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:04.286079 19756 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:04.287182 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:04.287348 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:04.287413 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:04.287536 19756 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:04.315207 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:04.316722 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.317827 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.318100 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:04.319361 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:04.402650 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:04.512230 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:04.569268 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (305.636s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:04.577759 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:04.608435 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:04.609665 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.611052 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.620126 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.627028 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:04.711736 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:04.717267 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.718794 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.721194 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.731052 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:04.761726 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:04.815999 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:04.819355 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:04.821135 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (305.886s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:04.821941 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:04.822170 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:04.822669 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:04.823899 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:04.825034 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:04.868335 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:04.876150 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:05.001144 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (306.068s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:05.053602 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:05.112495 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:05.114060 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:03:05.118199 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.118273 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:05.119508 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.121308 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.133296 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.208974 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (306.274s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:05.211045 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:05.212837 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.216164 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.220897 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.222157 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:05.224654 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:05.289641 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:05.306718 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 20:03:05.308482 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.310065 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.314718 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:05.315200 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.323254 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.415418 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (306.482s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:05.416281 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:03:05.469157 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:05.522274 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:03:05.559166 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:05.604643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.604667 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.607901 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.614624 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:05.615140 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.709450 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.710055 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.710814 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.712874 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:05.720732 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:05.728277 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.803750 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:05.809015 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:05.811594 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.811949 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.811959 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:05.822269 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (306.887s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:05.823127 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:05.837690 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (306.905s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:05.838129 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:05.944150 19800 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:05.944293 19800 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:05.944769 19800 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:05.946277 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:05.946357 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:05.946458 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:05.946581 19800 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:06.105742 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.111056 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.111330 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.114014 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:06.117614 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:06.138746 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:06.191147 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:03:06.201336 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.207423 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.210263 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:06.217301 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (307.284s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:06.219105 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (307.284s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:06.219934 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:06.220422 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.222047 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:06.248821 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:06.303573 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.305028 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.305262 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:06.306771 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.307302 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:06.308889 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:03:06.469991 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:06.598429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.599160 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.599846 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.602085 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.618588 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:06.638202 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:06.646711 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (307.712s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:06.647591 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:06.695466 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.696614 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.700150 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:06.701059 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.701436 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.745474 19800 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (307.812s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:06.786978 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:06.788540 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.789901 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.790946 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:06.797818 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:06.875300 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:03:06.944247 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:03:06.952229 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:06.956691 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:03:07.013957 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:07.081813 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:07.093930 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:07.094303 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.094565 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.107321 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.185004 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:07.185092 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.188441 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.189678 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:07.191154 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.197304 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (308.263s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:07.198848 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:07.215041 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:07.243573 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (308.311s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:07.273098 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:07.290683 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.293207 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.293336 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:07.295001 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:07.295940 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:07.297235 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:07.494035 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:03:07.579180 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 20:03:07.594201 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.594810 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:07.601970 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.602564 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.617555 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:07.618595 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:03:07.626649 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:07.669667 18852 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:07.669770 18852 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:07.670169 18852 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:07.671175 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:07.671317 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:07.671391 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:07.671505 18852 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:07.708428 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.709240 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:07.712246 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.712255 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:07.723572 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.734020 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (308.801s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:07.735368 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 20:03:07.799777 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (308.865s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:07.800493 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:07.802865 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.803329 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.803681 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:07.806835 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:07.819576 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:08.078155 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 20:03:08.106374 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.107003 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.111444 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.111515 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:08.125341 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:08.139778 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25ea8d7ab1834abfa34ff2357aa554b0: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:08.147433 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:08.200753 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.206496 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.206945 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:08.210029 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.213511 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (309.279s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:08.216563 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:08.231645 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:08.247095 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:08.306356 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.306376 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.309476 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (309.377s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:08.311578 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.319705 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.321544 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:08.327476 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:08.606840 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.608503 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.610121 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.610827 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:08.612385 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.625224 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (309.690s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:08.626415 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:08.638952 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:08.646660 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:03:08.667591 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:08.711704 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.712249 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:08.713996 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:08.714337 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:08.717082 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:08.726828 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:03:08.735363 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.812531 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.814728 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:08.817106 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:08.819123 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:08.823174 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (309.890s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:08.826686 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:08.992547 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 20:03:09.074573 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:09.121484 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.125245 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.126152 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.127729 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:09.137171 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (310.202s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:09.139236 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:09.140152 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:09.187913 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:09.208308 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:09.219898 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.221112 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.225131 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.225301 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:09.234133 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:09.234169 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:09.316087 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.318629 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.319386 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.319656 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.321627 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:09.440924 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (310.508s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:09.580217 10828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 26 similar messages]
W20240501 20:03:09.580945 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:09.607168 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.611757 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.616767 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:09.617774 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.618697 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.666818 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (310.732s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:09.667730 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:09.690610 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:09.711989 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.717041 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:09.717242 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.717934 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.718602 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.807003 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.809283 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:09.809450 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.810055 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:09.812470 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:09.827922 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:09.884569 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:09.891130 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:09.984097 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:10.001222 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (311.068s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:10.102094 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:10.106935 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.107023 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.113070 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.114835 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:10.127418 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.207299 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.208329 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:10.208617 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.210561 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.213747 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:10.217123 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:03:10.249487 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (311.315s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:10.250473 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:10.304293 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.304591 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.305526 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.309161 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:10.325866 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:10.401098 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 20:03:10.483489 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (311.551s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:10.484421 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:03:10.488895 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:10.598132 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:10.602886 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.605266 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.606810 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:10.607046 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.609197 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.677603 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (311.743s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:10.678575 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:10.705158 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.706332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.710606 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:10.717099 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:03:10.719107 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.719787 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:10.802161 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:10.803689 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240501 20:03:10.804811 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.805299 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:10.806847 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:10.818714 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:10.882982 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:10.912215 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (311.979s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:11.007133 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:11.105043 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.105971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:11.115070 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:11.118283 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.128051 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:11.167013 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:11.173555 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (312.239s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:11.174432 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:11.213614 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.214246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:11.219081 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.228287 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:11.229003 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:11.272845 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:03:11.305966 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:03:11.311233 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:11.314150 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.315263 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.324748 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.329028 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:11.467805 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (312.535s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:11.514201 21343 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:11.514282 21343 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:11.514595 21343 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:11.515411 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:11.515480 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:11.515525 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:11.515655 21343 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:03:11.559787 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:03:11.610646 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:11.612484 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:11.614686 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.615908 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.617640 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.617967 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:11.677665 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (312.743s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:11.678545 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:11.711339 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:11.714895 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:11.717859 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.718940 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.722904 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:11.734184 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.813122 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.814636 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:11.816006 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:11.819635 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.819640 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:11.825073 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:11.830000 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:12.014737 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (313.082s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:12.121978 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.123679 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.127091 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.133083 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.137203 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:12.173429 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:03:12.180722 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:12.224738 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.225796 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.227553 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.228986 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:12.238062 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.270583 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (313.336s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:12.271687 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:12.319197 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.319300 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.322458 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.322825 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:12.333632 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:12.333758 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:12.502902 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (313.570s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:12.513290 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:03:12.551004 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:03:12.603150 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:12.618822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.620560 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.620935 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.630308 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.631862 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:12.723703 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.724442 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.726043 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.726516 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.726547 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:12.775962 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:03:12.815403 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:12.819311 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.819840 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.825547 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:12.827602 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:12.832307 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (313.897s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:12.833540 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:13.036517 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (314.104s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:13.037456 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:03:13.106729 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 20:03:13.118259 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:13.121613 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.121866 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.123015 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.133424 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:13.184132 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:13.188863 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:13.220451 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:13.222704 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.226254 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.233484 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.235862 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:13.264022 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:13.312175 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 20:03:13.324172 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:13.326618 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.330307 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.331163 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.333189 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:13.337725 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (314.403s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:13.345480 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:13.446373 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (314.513s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:13.514246 22915 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:13.514339 22915 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:13.514628 22915 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:13.515628 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:13.515720 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:03:13.515779 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:13.515965 22915 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:03:13.595561 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:03:13.617249 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:13.624439 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.627893 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:13.628643 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.633054 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:13.646137 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.718912 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (314.784s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:13.719887 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:13.727586 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:13.729427 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.731949 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.732717 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.744565 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:13.794584 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:13.820355 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.823390 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.824291 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:13.824479 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:13.829031 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:13.875859 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:13.960832 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (315.028s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:14.062439 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:03:14.063851 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:14.122157 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.123641 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.123958 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:14.133816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.133951 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:14.134230 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:03:14.218988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.223999 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.225052 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.225813 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:14.225919 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.292899 18852 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (315.358s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:14.293788 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:14.312579 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.315227 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.315644 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.316581 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.331126 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:14.336155 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (315.403s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:14.351636 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:14.419925 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:14.450798 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:14.609720 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.609874 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.610450 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.612516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.613679 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:14.661265 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:14.669843 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:14.712675 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.712682 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.715224 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.722468 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.729516 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:14.733613 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (315.799s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:14.737649 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:14.778218 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:14.806542 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:14.812712 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.814900 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.815418 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:14.821349 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:14.894750 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (315.962s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:14.934732 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:15.107599 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:15.114627 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.115953 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.118342 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:15.119145 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.181488 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:03:15.182487 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:15.183403 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:03:15.211723 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:15.215322 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.218406 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:15.227046 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.233566 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:03:15.234365 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.280236 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:15.311623 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:15.311662 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (316.377s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:15.312197 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.313499 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.313634 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:15.318333 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.319075 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (316.386s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:15.320288 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:15.322875 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 20:03:15.333246 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:15.614320 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:15.618350 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.620256 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.621044 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:15.625061 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.639279 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:15.720583 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.722643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:15.722661 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.726014 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:15.737311 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:15.754488 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:15.803452 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (316.869s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:15.804487 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:15.812713 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.815790 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.816251 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:15.817039 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:15.819025 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (316.886s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:15.828588 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:15.880041 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:03:15.881734 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:03:15.905050 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:15.961613 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:03:16.066409 21343 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:16.066521 21343 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:16.066903 21343 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:16.067888 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:16.067970 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:16.068048 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:16.068164 21343 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:16.115705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.115726 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.115975 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.117483 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:16.137266 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:16.138926 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:16.214844 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.219961 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:16.220386 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.221850 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.229869 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.305325 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.307603 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.310432 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.319105 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.325176 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:16.340307 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25ea8d7ab1834abfa34ff2357aa554b0: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:16.352010 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:16.411500 23818 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (317.477s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:16.412252 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:16.422752 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (317.490s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:16.454011 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:03:16.606148 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.607954 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.609365 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:16.609606 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.618280 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.699405 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.702423 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.703615 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.707298 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:16.715108 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:16.754393 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:03:16.754834 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:16.799795 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:16.801313 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.804109 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:16.815199 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:16.824168 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:16.835093 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:03:17.001456 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:17.006351 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (318.073s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:17.022603 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (318.088s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:17.023499 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:17.061666 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:17.110035 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.110040 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.111713 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.115505 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:17.118039 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:17.156814 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:17.201694 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.204078 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.204293 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.207217 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:17.217574 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:17.233371 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:17.296095 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:17.311641 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.312963 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.313024 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:17.314599 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.322762 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:17.346010 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:17.485685 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (318.551s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:17.486502 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:17.515991 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (318.583s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:17.610587 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.612162 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.615710 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:17.617350 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.633399 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:17.684455 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 20:03:17.711402 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.713465 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.713805 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.722895 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.725301 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:17.805567 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:17.810393 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:17.812312 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:17.812822 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.813859 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:17.814520 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:17.844856 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:17.925026 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:03:17.930583 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (318.998s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:17.931392 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 20:03:17.956583 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (319.022s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:17.957445 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:18.111392 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.113134 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.116019 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.116648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:18.117318 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:18.153735 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:18.201957 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.202499 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:18.203691 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.208858 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.221796 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:18.294251 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:18.299834 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:18.302829 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:18.304260 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.303532 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.304826 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.422886 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (319.490s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:18.462910 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:18.519199 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:18.542732 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (319.608s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:18.546324 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:18.546813 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:18.582268 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 20:03:18.599977 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:18.601074 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:18.601672 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.601847 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.604365 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.696833 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.697480 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:18.710026 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:18.710031 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.712523 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.792994 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:18.795826 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.797076 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:18.799854 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:18.809330 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:18.920861 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:18.961622 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (320.029s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:19.028031 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:03:19.036325 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:19.094614 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.096381 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.099004 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:19.099901 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.106408 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:19.158648 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (320.224s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:19.159058 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 20:03:19.159586 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:19.195921 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:19.197037 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.198029 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:19.201067 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.222609 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.300519 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:19.301724 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.302551 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.303056 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.305071 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:19.438066 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:19.483467 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:19.553225 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:19.565245 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:19.579046 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:19.587074 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (320.654s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:19.593432 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:19.597553 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.599903 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.603526 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:19.603615 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:19.678057 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:03:19.694329 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:19.694900 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.698310 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.703872 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:19.704720 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.778231 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (320.843s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:19.779840 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:19.790796 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.795045 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:19.796414 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.796658 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:19.808648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:19.922175 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:19.939491 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:20.031245 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:20.032511 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (321.100s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:20.098098 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.099550 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.101359 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:20.116326 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.117748 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.197010 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.198943 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.199445 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:20.202008 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.205164 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.256708 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (321.322s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:20.257675 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:20.276898 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:03:20.295044 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.295938 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.299213 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.301185 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.301393 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:20.348599 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240501 20:03:20.519944 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (321.587s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:20.520751 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 20:03:20.592588 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:20.593027 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.596436 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:20.599412 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:20.601260 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.601876 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.668962 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (321.734s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:20.669874 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:20.688323 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.690199 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:20.692520 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:20.701799 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:20.702284 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.705845 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.784440 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:20.786163 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.787022 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:20.797199 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:20.799599 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:20.963099 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:20.986814 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:03:21.037909 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:21.048604 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (322.114s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:21.049567 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:21.077167 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240501 20:03:21.085901 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.087388 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.099256 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.105531 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.109588 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:21.113153 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:21.114158 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (322.181s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:21.185140 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.188800 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.190444 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.192168 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.205693 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:21.266534 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:21.290083 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.291322 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.291776 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.294605 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.305975 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.531337 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (322.597s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:21.532116 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:21.588016 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.588325 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.590008 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.591356 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.591591 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.675268 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (322.742s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:21.681722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.683681 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.685065 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.685598 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.697508 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:21.699379 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:21.713728 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:03:21.777524 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:21.778434 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.780966 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:21.783357 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:21.785077 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:21.795311 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:21.795787 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:21.901378 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:22.068948 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.071611 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.072255 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.072453 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:22.079393 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (323.145s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:22.081122 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:22.081318 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:22.102952 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:22.122393 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (323.189s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:22.146306 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:22.171921 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.175562 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.175843 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.181393 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:22.182108 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.273437 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.274820 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.275404 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.275992 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.278139 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:22.328127 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:22.339260 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:03:22.521744 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:22.537106 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (323.604s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:22.555095 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:22.567481 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.567559 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.571956 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.577428 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.577596 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:22.587460 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (323.653s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:22.588742 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:22.595386 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:22.601593 21343 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:22.601675 21343 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:22.601999 21343 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:22.602803 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:22.602883 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:22.602946 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:22.603111 21343 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:22.665431 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.669593 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.671921 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:22.677512 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.697393 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.768679 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:22.769075 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.770895 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.773432 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:22.776374 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:22.789060 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 20:03:22.942394 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (324.009s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:22.943269 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:03:22.963487 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:22.979161 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:03:23.067644 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:23.068143 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.069221 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.070497 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.071882 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:23.117041 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (324.182s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:23.117923 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:23.173122 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.174132 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:23.175534 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.176609 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:23.183898 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:23.216289 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:23.268249 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:23.269229 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.272076 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:23.273134 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.281530 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:23.452265 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:23.462327 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:23.488070 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (324.555s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:23.495062 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (324.560s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:23.495770 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:23.548799 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:23.575479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:23.576750 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 20:03:23.578254 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.581775 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.581928 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:23.582155 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.680759 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.682085 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.684799 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.685083 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:23.697191 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:23.775996 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:23.777045 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.777894 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.778383 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:23.779356 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:23.919106 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:23.945969 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (325.013s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:24.010187 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (325.075s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:24.010908 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:24.012658 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:24.072183 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.073915 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.081468 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.083204 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.083796 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:24.151806 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:24.174268 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.174794 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.177354 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.178041 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.178675 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:24.192930 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:03:24.255893 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:03:24.267323 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 20:03:24.268802 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.273221 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.273510 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:24.274747 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.276319 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.366842 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (325.434s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:24.526860 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:24.567616 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.570046 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.571604 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.572458 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:24.577728 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:24.578080 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:24.597795 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (325.663s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:24.599426 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:24.599692 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:24.655910 25223 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:24.656000 25223 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:24.656407 25223 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:24.658196 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:24.658386 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:24.658454 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:24.658653 25223 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:24.668586 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.669159 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.677978 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.686534 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:24.686913 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:24.708444 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:03:24.759332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:24.761397 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.762712 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.770530 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:24.771222 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:24.814280 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:24.831825 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (325.899s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:24.946300 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:03:24.976429 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:03:25.021708 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:25.025733 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (326.091s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:25.026706 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:25.035785 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:25.061211 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.061409 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.061939 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.065863 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.066327 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.165107 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.167804 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.168483 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.168632 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.170653 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.212702 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (326.280s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:25.213590 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 20:03:25.256091 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.259346 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.264531 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.265545 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.266255 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:25.432633 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:03:25.452673 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (326.518s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:25.453507 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:25.512041 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:25.554595 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.554982 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.558048 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.558722 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.573644 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.649652 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.651582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.654044 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.666831 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.675904 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.697748 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (326.765s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:25.752136 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:25.753094 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.755793 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:25.757546 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:25.757792 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:25.938503 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:25.938585 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:25.985554 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (327.051s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:25.986361 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:25.991278 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:03:26.058529 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.060930 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.062345 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.064711 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:26.065621 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.078706 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (327.146s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:26.085384 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:03:26.093112 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240501 20:03:26.158804 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.161129 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.161378 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:26.162107 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.162791 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:26.204787 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:26.248867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.249673 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.251418 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.253067 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.258225 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:26.316082 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:26.413733 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:26.464597 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (327.532s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:26.467334 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (327.533s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:26.468067 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:26.518836 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:26.543643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.544009 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.546466 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.549042 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.551285 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:26.635615 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.638669 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.641052 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:26.645674 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:26.646540 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.669996 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:26.742519 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.744832 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:26.745219 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:26.745561 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:03:26.753649 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:26.756004 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:26.793395 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:03:26.934073 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:26.965309 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (328.031s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:26.966114 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:26.989001 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:27.005792 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (328.073s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:27.036654 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.038755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:27.039186 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.041013 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.054256 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:27.130360 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:27.131552 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.135052 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.137640 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.149202 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:27.199476 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:27.230926 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.232556 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.233987 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:27.241145 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:27.244725 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.245008 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:27.358690 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:03:27.361189 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 20:03:27.427605 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (328.493s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:27.428479 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:27.478947 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:27.526477 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:27.528676 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:27.530875 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:27.531580 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.533196 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (328.600s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:27.534083 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240501 20:03:27.534188 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.537294 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:27.621786 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:27.622143 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.623212 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.626859 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:27.634124 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.720460 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.722131 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.723882 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:27.725207 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:27.735725 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:27.962540 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (329.028s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:27.963557 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:28.014022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.015877 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.018545 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:28.018909 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.029117 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:28.041903 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:28.113907 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.117518 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.118636 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:28.120852 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.121297 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.145958 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (329.213s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:28.203188 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.212443 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.213085 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.213955 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:28.225195 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:03:28.226508 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:28.393479 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:03:28.395522 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:03:28.413830 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:28.508433 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.508659 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.509059 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.510605 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.511543 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:28.560526 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:28.572167 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (329.637s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:28.573194 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:28.596927 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.600935 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.607756 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.608493 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.609253 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:28.691351 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.693989 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.695456 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.695580 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:28.704627 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:28.745080 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:03:28.762449 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:28.765731 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (329.833s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:28.911087 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:28.985868 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.986963 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:28.987553 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:28.999699 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.006564 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:29.012290 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:29.045382 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (330.111s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:29.046149 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:29.078995 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.085116 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:29.088105 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.088356 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:29.102967 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:29.106766 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.182536 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.184919 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.185319 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.186506 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.188225 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:29.230684 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (330.298s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:29.254091 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:03:29.341888 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:03:29.362010 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:03:29.472128 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:03:29.480876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.484738 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.485219 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.491120 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (330.556s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:29.492208 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:29.492493 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:29.503903 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.575346 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.580722 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.581573 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:29.583647 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.585376 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:29.597213 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:29.677229 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.679622 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.681519 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.682564 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:29.682754 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:29.691097 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:03:29.733454 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:29.745512 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (330.813s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:29.802919 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:03:29.829208 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:29.957468 26703 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:29.957580 26703 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:29.957948 26703 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:29.959915 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:29.960235 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:29.960295 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:29.960430 26703 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:29.974745 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.977422 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.977699 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:29.978734 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:29.984843 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:29.985194 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:30.008999 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:30.039119 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (331.104s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:30.039937 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:30.070542 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.070935 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.071911 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.071947 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.075975 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.163702 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.167963 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.169631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.169872 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.173081 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.218082 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (331.285s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:30.218942 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 20:03:30.441206 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:03:30.458907 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.462903 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.463718 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.466487 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.466954 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.557852 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.562146 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.562513 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.565693 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.570195 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:30.604779 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:30.641582 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (331.707s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:30.642483 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:30.653085 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.657581 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.664158 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.666962 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.679196 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.694321 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (331.761s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:30.833266 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:03:30.883760 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:30.953851 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:30.954157 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.955355 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:30.958531 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:30.963359 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:31.013089 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:03:31.018682 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:31.048651 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.049211 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.049424 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.052556 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.057072 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:31.063498 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (332.129s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:31.066751 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:31.145840 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:31.146854 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.149708 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.153062 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:31.153228 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.153604 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:31.275583 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:31.311594 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (332.379s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:31.443711 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.444365 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:31.446098 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.446730 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.449580 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:31.469540 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:03:31.491997 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:03:31.520936 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:31.535068 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.538635 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.539352 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.546255 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.552101 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:31.589125 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (332.654s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:31.589990 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:31.605975 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:03:31.606628 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:31.612802 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:31.634428 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.634510 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.635890 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:31.638689 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.641031 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:31.736610 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:03:31.760777 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:03:31.808380 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (332.875s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:31.873320 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:31.937494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:31.938298 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.941061 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.948927 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:31.950328 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.026954 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:32.028782 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:32.030208 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.030371 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.031069 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.043704 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.127651 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:32.130874 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.132519 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.134208 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:32.144748 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:32.147887 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.157943 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (333.223s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:32.158927 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:32.262269 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:03:32.285418 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:03:32.326856 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:32.424312 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.428565 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.429433 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (333.496s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:32.435892 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:32.436566 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.440698 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:32.509506 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:32.524103 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.524137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:32.527751 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.528131 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.530792 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:32.575615 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:32.580893 27648 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:32.581046 27648 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:32.581415 27648 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:32.582458 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:32.582537 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:32.582595 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:32.582733 27648 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:32.620245 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.620616 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:32.622817 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.625605 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.627169 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (333.692s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:32.628266 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:32.628567 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:32.739362 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25ea8d7ab1834abfa34ff2357aa554b0: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:32.923429 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.925688 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.926523 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:32.928092 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:32.928730 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:32.937230 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (334.004s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:32.939992 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:03:32.977768 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:33.012879 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.017055 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.019340 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.019539 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.019950 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.125083 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.127454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.127671 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.130259 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.133188 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.175534 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (334.241s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:33.176432 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:33.254086 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:33.371924 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:33.419765 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.424672 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.429028 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.431254 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.433773 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.523643 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.524513 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.524559 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.525094 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.526170 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.537812 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (334.605s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:33.587213 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 20:03:33.591655 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (334.657s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:33.592762 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:33.623276 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.624246 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.624572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.625677 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.626341 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:33.639997 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:03:33.663280 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:03:33.696751 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:33.780512 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:33.919736 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.920285 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:33.920642 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:33.922513 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:33.929988 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:33.948469 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:03:34.008402 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:34.011907 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:34.017078 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:03:34.018646 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:34.019088 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.021068 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:34.080374 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:34.081135 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (335.148s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:34.112421 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.112838 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.125737 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.126942 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:34.130220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:34.149665 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:34.181809 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (335.247s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:34.182715 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:34.209331 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:34.303575 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:34.350031 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:03:34.387836 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:34.409691 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.410874 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:34.411573 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.413393 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:34.422209 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:03:34.425341 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:34.503623 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:34.504850 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.505025 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.513967 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.523519 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:34.548180 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:03:34.594076 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (335.661s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:34.599169 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:34.601130 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.603101 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.605084 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.609068 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:34.693804 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (335.759s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:34.694550 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:34.740044 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:03:34.850176 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:03:34.901166 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.901806 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.902192 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:34.904179 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.913946 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:34.914017 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:34.976529 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:34.980619 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:34.997090 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.999786 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:34.999814 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.001351 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.009058 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.045185 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (336.112s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:35.093102 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.096390 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.096580 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.103451 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.106093 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.251430 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (336.317s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:35.252276 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:35.382520 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:35.392489 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.394292 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.397127 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.407083 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.413400 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.487179 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.488780 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.490574 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.492143 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.498159 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.533422 28033 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:35.533515 28033 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:35.533862 28033 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 20:03:35.536613 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (336.604s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:35.539769 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:03:35.539871 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:35.539925 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:35.540048 28033 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:03:35.540578 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 20:03:35.580790 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.581038 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.588109 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.593322 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.593850 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:35.610332 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:03:35.871490 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (336.937s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:35.872306 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:35.882494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.884842 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.886880 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.889150 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.897404 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.969532 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (337.037s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:35.975364 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:35.976204 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:35.978479 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.979015 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:35.984437 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:36.007328 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:36.055572 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240501 20:03:36.074518 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.078908 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.087604 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.089638 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.095438 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:36.117130 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:03:36.210881 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:36.333709 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:36.368554 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:36.385732 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:36.387454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.387511 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.391255 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:36.391645 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.401499 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:36.422752 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:36.439623 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:36.445484 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (337.511s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:36.446687 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:36.454193 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (337.521s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:36.481125 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.486135 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.486330 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.488440 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:36.488867 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.573354 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.574896 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.576663 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.577060 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.577419 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:36.605975 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:36.791599 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:36.807061 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:36.819391 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:36.872808 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.875489 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.876036 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.877377 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.892957 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:36.900045 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (337.965s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:36.900925 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:36.969050 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.971865 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:36.972311 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.972570 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:36.984426 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:37.021857 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:37.059269 21343 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (338.126s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:37.066076 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.069352 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.072773 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.073351 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:37.074827 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:37.101130 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:03:37.216547 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:03:37.216985 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:37.294121 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:37.369186 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.369262 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.372269 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.381021 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.382061 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (338.447s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:37.383764 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:37.397017 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:37.429062 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:37.448824 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (338.516s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:37.472193 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.476887 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.477634 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:37.480683 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.485975 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.567456 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.569283 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.572619 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:37.573341 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.585723 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:37.615270 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:37.682233 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:37.765666 23948 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:37.765774 23948 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:37.766183 23948 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:37.767335 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:37.767414 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:37.767473 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:37.767597 23948 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:37.869550 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.875342 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.877401 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.882700 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:37.889117 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.951484 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (339.017s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:37.952378 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:37.972257 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:37.978683 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.979086 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.979736 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:37.980351 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.057037 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (339.124s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:38.058044 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 20:03:38.068076 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.069031 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.081470 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.081620 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.081712 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:38.088418 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:38.342594 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:03:38.369975 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.373531 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.376622 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.382067 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.389900 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.421471 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (339.487s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:38.422262 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:38.463689 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.464671 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.465659 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.467512 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.477270 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (339.544s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:38.478586 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.554723 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.556316 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.557804 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.559111 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.573562 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:38.625492 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:38.637382 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:03:38.709872 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:38.795874 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:38.851168 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.852424 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.854089 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.855136 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.867000 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.880801 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (339.946s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:38.881906 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:38.910831 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (339.978s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:38.952147 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:38.952678 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:38.953572 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:38.954246 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:38.957100 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:38.963562 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:38.964540 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:38.978284 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:39.049885 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.050595 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.051678 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.051949 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:39.071552 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:39.081573 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:03:39.083549 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:03:39.139950 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:39.299456 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (340.366s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:39.310490 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:03:39.322127 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (340.387s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:39.323115 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:39.353768 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.354092 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:39.359508 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.365835 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.366369 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:39.375177 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:03:39.448890 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 20:03:39.451398 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.451687 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.454151 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:39.456660 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.461066 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:39.479981 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:03:39.545755 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:39.548686 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:39.549875 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.553647 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:39.555809 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.568012 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.676117 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (340.743s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:39.680649 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240501 20:03:39.841217 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (340.906s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:39.842830 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:39.853914 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.853914 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.861668 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:39.863410 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.863982 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:39.940033 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:39.950719 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:39.951037 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:39.953863 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:39.955471 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.956290 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:39.958264 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.042029 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.044430 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.049085 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.049362 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:40.050105 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.212236 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (341.279s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:40.213217 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:03:40.243230 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:40.283046 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (341.348s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:40.283892 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:40.309842 23948 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:40.309952 23948 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:40.310314 23948 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:40.311239 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:40.311316 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:40.311375 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:40.311507 23948 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:03:40.322755 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:40.345000 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.348114 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.349110 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.351819 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.371764 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:40.446731 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.447815 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.449067 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.451766 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:40.457180 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:40.528115 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:40.538681 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.543588 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.545964 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.547698 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:40.548314 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.619760 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (341.687s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:40.691644 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:03:40.824188 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (341.889s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:40.827086 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:40.828889 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240501 20:03:40.833176 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.839076 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.840173 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.842041 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.842115 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:40.928030 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:40.930778 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.931465 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.932415 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:40.933449 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:40.966975 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:41.019877 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.023021 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.023119 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.024431 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.028439 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.089756 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (342.157s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:41.234668 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (342.300s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:41.235569 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:41.270054 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240501 20:03:41.320816 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.321446 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.321949 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.327582 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.330865 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:41.402323 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:41.415768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.418887 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.419431 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.420305 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.425417 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:41.444904 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:41.479480 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:41.508092 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.510295 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.510634 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.512161 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.513056 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:41.539002 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:41.552577 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:03:41.612918 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:41.666167 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (342.733s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:41.667924 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:03:41.737655 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (342.803s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:41.738442 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:41.754844 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:03:41.762907 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:41.773350 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:41.781086 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:41.809096 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.811645 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.812546 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.813059 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.814188 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:41.828743 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:41.901479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.901854 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.906999 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.909318 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:41.915124 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:41.997311 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:41.998946 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.003595 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.004225 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.006155 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.099113 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (343.166s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:42.128341 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (343.194s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:42.129382 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:42.256683 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:03:42.281623 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:42.301544 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.302035 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.302181 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.304149 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.304986 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.392633 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.393275 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.395294 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.395606 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.402993 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.484786 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.484898 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.485424 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.487444 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.497772 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:42.536165 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:42.554311 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (343.621s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:42.555088 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:03:42.612502 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:42.654163 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (343.719s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:42.655138 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:42.730934 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:42.747757 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:42.780241 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.782327 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.784448 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.785642 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.787169 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.879683 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.880378 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.882210 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.885010 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.886706 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.975826 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:42.976598 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (344.044s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:42.977614 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.983378 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:42.987851 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:42.995229 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.080425 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (344.146s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:43.081327 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:43.155982 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:43.285395 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.293788 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:43.294289 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.295866 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.295878 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:43.371115 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:43.381593 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240501 20:03:43.390367 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.391510 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.393229 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:43.394068 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:43.394076 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.483011 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:43.490803 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (344.556s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:43.491797 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:43.503489 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.504384 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.505600 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:43.506770 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.587224 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (344.654s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:43.770071 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:43.795490 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.798745 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:43.799027 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.799229 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.799286 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:43.816208 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:43.827073 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:43.828979 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:43.893652 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:43.897828 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:43.898197 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:03:43.900555 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.901206 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:43.910261 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (344.975s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:43.911623 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:43.919518 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:43.999719 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.001000 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:44.001142 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.002230 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.009052 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.041710 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (345.109s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:44.044008 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:44.184401 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:44.243247 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:44.267791 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:03:44.282127 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:03:44.293548 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.297536 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.297777 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:44.299585 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.316270 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:44.372679 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:44.396502 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.397295 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.399096 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.400564 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.401046 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:44.448765 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:44.467188 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 20:03:44.496258 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.496367 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.505988 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.506359 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.506655 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:44.534391 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (345.600s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:44.535392 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:44.579981 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (345.647s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:44.629506 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 20:03:44.793582 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.796501 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.801069 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:44.802338 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.804008 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.894829 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:44.896116 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.898173 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.906000 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.908622 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:44.914985 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:44.990123 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (346.057s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:44.991019 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:44.992058 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240501 20:03:44.992319 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:44.994405 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.002633 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.002897 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.044469 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (346.110s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:45.045365 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:45.127231 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:45.195381 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:45.210467 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:03:45.291867 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:45.294488 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.294879 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.296866 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.297026 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:45.340837 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:45.390787 30202 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:45.390898 30202 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:45.391306 30202 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:45.392140 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:03:45.392199 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.393388 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:03:45.395562 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:45.395629 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:45.395888 31295 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:45.396245 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.402611 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.408571 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.485580 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (346.551s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:45.486732 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:45.488590 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:45.491026 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.493450 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.493700 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.494813 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.533597 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (346.601s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:45.659376 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:03:45.708237 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:03:45.784832 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.789222 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.793049 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.797757 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:45.801177 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:45.889075 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:45.889346 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:45.889699 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.894678 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.895278 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:45.905282 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.987706 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.990188 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:45.991340 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:45.993995 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.003218 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.016891 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (347.084s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:46.099189 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (347.164s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:46.100152 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:46.179225 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:46.227751 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:46.272073 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:46.290519 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.292383 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.292833 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.293660 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.295109 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:46.305423 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:03:46.385636 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.387073 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.388383 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.398939 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.399181 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.479723 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.484786 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.489558 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.492698 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.493570 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.532193 23948 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (347.599s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:46.555562 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:03:46.566744 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 20:03:46.601720 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:46.656296 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:46.663581 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (347.729s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:46.664446 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:46.675727 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:46.681633 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:03:46.780180 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.780718 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.782112 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.784355 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.785331 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:46.806139 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:46.825855 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:46.884083 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.886571 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.888216 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.889834 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.901074 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:46.913686 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 20:03:46.929311 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (347.996s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:46.930428 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:03:46.980412 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.980988 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:46.983597 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:46.985426 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:46.989406 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.069929 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (348.135s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:47.070847 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:47.145951 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:47.279227 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.279588 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.289729 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.289729 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.290494 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.374876 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.375602 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.376499 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.379750 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.387113 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:47.451889 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:47.471561 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.471596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.475430 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.476214 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.477283 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.502888 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (348.570s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:47.503813 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 20:03:47.536064 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (348.601s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:47.537176 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:47.624934 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:47.691701 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:47.701457 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:47.777295 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.780817 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.783197 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.790091 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.798220 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.885946 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.886165 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.889240 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.890111 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.898358 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.983110 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:47.983296 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.983577 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:47.990303 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:47.991211 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:48.059370 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:48.089808 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240501 20:03:48.090405 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (349.155s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:48.091347 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:48.124181 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (349.191s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:48.281672 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.282581 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.285130 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.286460 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:48.297039 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.374687 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.377311 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.377777 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.380652 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.383014 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:48.473429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.475522 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.477155 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.478602 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.485731 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:48.582365 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:03:48.582912 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:48.592676 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:48.599591 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:48.637207 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (349.702s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:48.638029 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:48.674561 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:48.713865 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (349.781s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:48.769918 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.771579 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.772821 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.774976 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.781955 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:48.868556 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.870513 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.872756 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:48.883699 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.884034 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:48.944043 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:48.967185 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.967852 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:48.972712 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.979046 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:48.989743 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.149752 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (350.215s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:49.150722 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:49.153534 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:49.208654 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:49.270275 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.271247 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.272835 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:49.279999 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:03:49.281777 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.288652 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:49.289078 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:49.291159 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:49.303040 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:49.322310 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 20:03:49.325042 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (350.392s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:49.373101 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.373137 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.374575 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.374894 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.393574 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.468050 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.472018 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.473031 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.476065 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.482678 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.564885 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (350.630s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:49.565280 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:03:49.565830 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:49.615556 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:49.697016 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:03:49.747015 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:49.770738 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.771356 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.784296 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.784879 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.786203 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.837929 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (350.905s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:49.882197 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.884413 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.888052 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.888446 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.897188 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.980690 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.980755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:49.982995 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:49.985769 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:49.995491 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.031944 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (351.097s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:50.033008 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:50.151973 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:50.198004 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:50.214561 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 20:03:50.276074 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.281002 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.281458 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:50.281692 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.283717 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.336169 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (351.403s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:50.337469 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 20:03:50.373256 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.373955 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.378409 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.390206 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.390868 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:50.461511 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:50.479266 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:50.479710 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.480158 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.480541 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.482082 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.643184 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (351.708s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:50.644019 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:50.665246 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:50.763134 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:03:50.771631 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.774045 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.775543 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.777222 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.787665 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:50.875052 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.878530 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.881544 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.884300 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (351.951s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:50.887171 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.889613 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:50.971446 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:50.975162 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.976241 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.977396 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:50.982815 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:51.101192 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:03:51.140256 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:51.174937 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:51.196722 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:51.241286 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (352.306s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:51.242353 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:51.247686 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:51.272755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.275969 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:51.277096 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.286098 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.286744 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.375586 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.376473 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.376885 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.387367 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.397761 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:51.409271 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:51.474596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.475096 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.478363 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:51.478474 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.495838 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (352.563s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:51.496230 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:51.523550 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:03:51.639030 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:03:51.689224 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:51.726794 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (352.792s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:51.727691 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:51.775460 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.778081 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.781126 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.789467 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:51.790138 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:51.800318 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:03:51.870087 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.871503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.876089 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.880697 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.892130 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:51.944337 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 20:03:51.970546 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:51.971465 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.972293 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:51.974102 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:51.982213 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:52.003839 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:03:52.008960 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:03:52.009363 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:03:52.012466 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (353.080s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:52.075585 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:52.267021 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.270814 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.271535 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.271800 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:52.278314 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:52.285399 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.331111 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (353.396s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:52.332420 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:52.362401 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.367097 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.367419 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.370810 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.379390 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.461205 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.462352 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.462848 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.463069 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.480165 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.566088 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (353.633s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:52.671650 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:52.757095 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.757488 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.760772 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:52.768599 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:52.769253 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.769654 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.808216 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (353.873s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:52.809263 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:52.810895 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:03:52.817127 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 20:03:52.851177 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.851547 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.854271 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.863482 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.866108 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.912882 29370 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:03:52.913048 29370 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:52.913389 29370 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:03:52.914462 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:52.914546 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:52.914608 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:03:52.914803 29370 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:03:52.954764 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.955303 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:52.955911 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:52.956869 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:52.969502 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.170554 31971 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (354.238s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:53.171515 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 20:03:53.194941 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (354.260s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:53.195982 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:53.252993 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.253468 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.262265 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.263931 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.270474 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:53.283552 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:53.358677 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.360698 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.362246 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.373692 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.376492 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:53.403403 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:53.455009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.457891 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.459558 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.470294 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.477654 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:53.627868 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (354.693s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:53.628753 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:53.738548 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:53.755309 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.757032 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:53.757251 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.759366 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.759431 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.770627 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (354.838s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:53.789224 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:53.850487 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.853590 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.863363 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.864787 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:53.866495 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:53.878050 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:53.959251 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:53.960597 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.962040 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:53.963467 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:53.976475 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:03:53.977196 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:53.978328 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:53.991926 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:03:54.043566 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:54.120144 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (355.185s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:54.121069 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:54.144340 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:54.259717 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.263774 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.264948 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.266068 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.269201 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:54.339979 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (355.407s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:54.357385 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.357515 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.358106 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.358389 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:54.358635 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:54.381422 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:54.381934 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:03:54.409050 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:03:54.419430 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:54.435020 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:03:54.460095 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.460410 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.460850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.464134 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.468210 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:54.536218 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (355.601s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:54.537081 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:54.619788 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:54.750254 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.754545 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.754757 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.756242 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.757859 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:54.762177 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:03:54.831986 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:54.849829 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.852167 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.852758 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.856168 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:54.857023 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.939950 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (356.007s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:54.941545 781 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (356.007s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:54.942459 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:54.952674 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:54.953097 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.957291 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.957710 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:54.963951 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:55.078804 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:03:55.230327 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:55.253938 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.255108 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.260815 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.267531 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:55.268390 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:55.280421 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 20:03:55.328176 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:55.356700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.359114 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.359202 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.361575 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.367993 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (356.433s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:55.369096 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:55.370234 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:55.453022 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.455678 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.455974 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.458060 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.460332 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:55.501087 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (356.568s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:55.748172 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.748572 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.751422 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.751811 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.760459 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:55.770890 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:55.823182 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:55.845676 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.846354 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.849706 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.850333 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:55.850378 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:55.885448 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:55.885538 781 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (356.951s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:55.886463 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:55.940814 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.941793 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:55.944331 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.955114 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:55.959723 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:56.034283 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:56.115962 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (357.183s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:56.116868 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 20:03:56.209223 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:56.250978 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.253432 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.253648 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.254649 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.262645 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:56.277307 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:56.342875 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.347414 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.347551 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.353080 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.357090 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.381668 781 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (357.447s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:56.382864 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:56.430425 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:03:56.442090 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.442845 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.445370 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.447603 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.455144 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.500361 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (357.567s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:56.621966 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:56.684312 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:56.733381 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:56.737468 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.740521 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.740744 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.741024 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.742775 31295 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:03:56.742862 31295 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:03:56.743188 31295 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
I20240501 20:03:56.749042 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:56.749650 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:56.749915 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:03:56.749997 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:03:56.750380 31295 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:03:56.799750 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:56.815301 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (357.880s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:56.816215 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:56.834535 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.834823 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.835650 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.837838 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.842051 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:56.876086 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:03:56.915251 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (357.982s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:56.925302 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:56.942974 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:56.944835 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.946038 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:56.954118 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:56.957088 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:56.979385 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:03:57.010012 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:03:57.095579 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:03:57.177778 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:57.244068 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.246246 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.253302 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.255245 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:57.255784 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.344053 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.346575 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.347957 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:57.350144 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.350560 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (358.418s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:57.363832 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.386061 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (358.451s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:57.386950 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:57.426453 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:57.443435 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:57.443922 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.444355 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.447115 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.461730 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:57.597447 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:57.740784 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.744027 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.744288 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.746827 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.747473 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:03:57.750918 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 20:03:57.770236 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:57.816305 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:57.838806 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.840127 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.840813 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.843278 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.866019 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:57.896708 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (358.962s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:57.897675 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:57.943619 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:57.944093 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.949276 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:57.949805 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.950584 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:57.973223 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (359.040s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:58.084949 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:58.244818 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.249169 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.250119 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.254566 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.264151 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.346966 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.351509 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.355518 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.356137 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.359321 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.384971 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (359.450s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:58.385918 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:58.395097 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:03:58.413777 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (359.481s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:58.414722 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 20:03:58.449456 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.453063 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.454996 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.455291 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.455847 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:58.647753 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:58.652753 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:03:58.748840 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.749359 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.749770 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.750663 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.766125 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.805032 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (359.870s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:58.806136 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:58.846679 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.847028 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.847347 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.847862 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.849972 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.888198 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (359.955s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:58.936841 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:58.941171 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:58.944515 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.946506 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:58.949539 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:58.961266 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:58.995780 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:03:59.054713 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:59.072898 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:59.103435 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:03:59.242357 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.243685 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:59.244341 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.253234 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.253464 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:59.254720 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:03:59.343243 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.346046 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.346815 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.349424 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:59.354087 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (360.419s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:59.355724 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:59.357986 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:03:59.364964 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:03:59.407683 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 20:03:59.437050 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.441061 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.443323 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.449661 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:59.450022 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.458215 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (360.525s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:59.553185 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:03:59.570750 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:03:59.581961 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:03:59.621073 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:03:59.692071 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:03:59.729627 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (360.795s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:03:59.730818 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:03:59.742586 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.742709 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.744385 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:59.744663 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.746897 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.835752 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.838842 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:59.839453 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240501 20:03:59.840163 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.841879 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:03:59.848794 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:03:59.867300 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240501 20:03:59.884670 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (360.952s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:03:59.937530 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.938367 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:03:59.942034 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.945194 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:03:59.952075 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:00.118430 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (361.184s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:00.119341 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:00.130713 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:00.236671 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.239053 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.241983 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:00.248227 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.253336 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.330399 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.330648 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.339879 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.341327 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.355991 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (361.423s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:00.357290 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:00.415205 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:04:00.440587 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.441061 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.441171 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.444044 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:00.453753 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:00.532953 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:04:00.558480 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (361.624s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:00.559266 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:00.663802 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:00.714598 2023 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:00.714699 2023 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:00.715085 2023 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:00.716758 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:00.716846 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:00.716958 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:00.717386 2023 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:00.737051 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.737327 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.741879 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.743253 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.745520 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:00.779021 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (361.846s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:00.780508 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:04:00.799758 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:04:00.831692 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.831709 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.832206 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.835394 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:00.843827 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:00.920195 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:00.925359 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:00.932657 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.933955 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.934962 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:00.937317 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:00.945127 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:01.042515 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (362.108s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:01.043409 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:01.044669 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:01.162648 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (362.230s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:01.234936 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.235587 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.239786 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.240068 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.247556 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:01.286805 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:04:01.328547 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.333967 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.335665 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.341056 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:01.341300 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.424917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.428773 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.436223 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.437162 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.438249 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:01.511179 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:04:01.534128 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:01.571590 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (362.637s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:01.572515 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:01.581929 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (362.649s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:01.594570 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:01.636772 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:01.723027 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.724499 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.727869 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.730032 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.738471 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:01.765069 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:01.819048 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.820853 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:01.822912 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 20:04:01.825378 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.826714 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.839232 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:01.919723 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.920351 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:01.922773 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.923230 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:01.929345 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:01.962548 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (363.030s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:02.029667 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (363.095s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:02.030092 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:02.030565 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:02.059967 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:04:02.116832 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:02.150635 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:04:02.165181 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:02.198966 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:02.214573 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.216595 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.216606 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.218679 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.228564 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.309950 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.310187 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.313903 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.315472 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.323519 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.383564 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (363.451s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:02.404189 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.405855 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.407523 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.407557 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.408213 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.421140 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (363.486s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:02.427230 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:02.543812 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:02.615412 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240501 20:04:02.702797 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.703336 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.705410 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.707427 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.712954 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.799585 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.800150 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.801381 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.801573 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.805181 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.812275 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (363.879s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:02.813326 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 20:04:02.894819 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.896636 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:02.899355 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:02.900166 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:02.907722 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:02.948798 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:02.950201 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (364.015s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:02.951052 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:02.973104 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:02.994256 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:03.192967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.195628 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.200129 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.205761 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:03.211290 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.232535 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (364.300s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:03.300694 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.301082 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.304710 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.305992 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:03.318464 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:04:03.321799 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.398662 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.398670 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.403081 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:03.407449 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.408465 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:03.432438 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:03.432636 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (364.498s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:03.433494 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:03.433975 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:03.631584 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:03.702062 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.707245 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.707366 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.710433 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:03.715929 2227 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:03.716024 2227 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:03.716296 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.716396 2227 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:03.717543 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:03.717696 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:03.717752 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:03.717926 2227 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:03.795418 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.796063 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:03.797549 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.797904 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.798748 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.809273 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (364.876s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:03.878145 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:03.889102 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.890039 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.890987 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:03.891455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:03.894050 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:03.945616 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:03.946991 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:03.955560 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (365.021s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:03.956388 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:04.073213 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:04.137930 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:04.182175 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.183403 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.184542 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.184679 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.189600 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.279088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.279134 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.280468 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.281875 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.285077 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:04.368866 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:04.370368 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.374171 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.374768 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.377661 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.380286 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.387596 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (365.453s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:04.396719 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:04.399840 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:04.404259 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:04:04.404883 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:04:04.423225 2023 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (365.490s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:04.471738 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:04:04.519299 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:04.571522 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:04.584102 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:04:04.621529 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:04.629508 2023 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:04.629606 2023 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:04.629942 2023 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:04.630863 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:04.630934 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:04.631018 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:04.631153 2023 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:04.676960 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.681286 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.681432 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.684952 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.698596 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.775503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.777724 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.778280 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.779636 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.788880 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.821456 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (365.889s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:04.861649 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 20:04:04.862250 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:04.872071 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.875764 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:04.875996 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.878311 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:04.887208 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:04.950960 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (366.016s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:04.951751 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:05.167507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.168267 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.171213 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.176347 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.181869 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:05.184862 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 20:04:05.197242 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (366.264s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:05.198729 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:04:05.262393 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:05.263783 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.266564 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.267067 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:05.268719 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.275231 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:05.350996 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:05.354949 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.355187 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.358134 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.360471 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (366.426s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:05.361447 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:05.361585 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:05.371405 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:05.534200 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25ea8d7ab1834abfa34ff2357aa554b0: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:05.603529 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:05.637249 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:05.661507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.663084 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.664177 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.665776 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.667577 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:05.698621 29370 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (366.766s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:05.759732 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.761868 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.764657 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.765461 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:05.772773 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.778142 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (366.843s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:05.778963 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:05.855755 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:05.857816 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.859746 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:05.860580 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:05.871455 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:06.071961 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:06.076052 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:06.139874 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:04:06.162156 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.162853 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:06.162919 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.163611 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:06.164414 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:06.173739 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:06.214036 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:06.215479 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (367.283s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:06.220702 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (367.286s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:06.223573 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:06.266376 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.270956 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:06.272270 2227 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:06.272361 2227 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:06.272454 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.272704 2227 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:06.274379 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:06.274662 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:06.274732 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:06.274987 2227 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:06.279824 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:06.289186 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.365473 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:06.369728 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.370299 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:06.371237 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.372097 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:06.382066 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:06.383387 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:04:06.524602 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:06.663120 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:06.669596 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.669816 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (367.735s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:06.670954 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:06.673965 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.684170 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:06.685765 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.712508 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (367.780s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:06.770351 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.772233 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:06.772715 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.773252 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.773749 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:06.804349 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:06.859318 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.862923 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:06.863463 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.867741 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:06.871333 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:06.899960 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:04:06.927758 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:04:06.941442 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:04:06.967046 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:04:06.983091 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:06.999084 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:07.070974 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240501 20:04:07.128289 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (368.193s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:07.129298 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:07.164620 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.164917 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:07.165885 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.167444 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.182255 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:07.267398 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:07.267985 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240501 20:04:07.268146 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.273689 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:07.273718 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.281852 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.283584 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (368.351s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:07.356791 4056 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:07.356921 4056 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:07.357254 4056 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:07.359422 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:07.360106 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:07.360177 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:07.362746 4056 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:07.364666 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.365252 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:07.365969 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.369098 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.377332 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:07.405344 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:07.482936 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:04:07.572950 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (368.638s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:07.573712 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:07.604543 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:07.662331 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:07.664016 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:07.666764 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 20:04:07.675575 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.675791 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.679018 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:07.739406 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (368.806s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:07.740681 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 20:04:07.761148 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:07.763684 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.764081 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.764668 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:07.785461 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:07.815742 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:07.858824 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.859797 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:07.866881 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:07.880363 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:07.881975 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:07.982370 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:08.106026 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (369.171s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:08.106987 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:08.162184 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:08.166435 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.167970 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.168628 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.174815 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:08.257349 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.259546 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:08.259917 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (369.327s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:08.260370 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.265852 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.277202 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:08.350486 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:08.352413 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.356968 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:08.359534 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:08.360173 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.362416 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.576843 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (369.642s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:08.577749 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:08.650094 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.651903 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.658411 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.669313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:08.669679 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:08.733646 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:08.754988 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.755120 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:08.756693 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.757596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:08.765647 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:08.765945 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:08.832450 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:08.845639 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:08.850874 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.852866 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (369.920s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:08.858004 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.861568 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:08.861742 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:08.873785 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:04:08.877677 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:09.117213 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (370.182s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:09.118243 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:09.149533 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.150157 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.152983 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.157310 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:09.158142 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:09.178742 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:09.244674 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.246981 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.251446 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.257926 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:09.263165 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:09.289204 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:04:09.313750 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:04:09.325625 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:09.329780 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:09.342572 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.346451 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.346982 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:09.354658 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.370048 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:09.378839 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:09.458283 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (370.525s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:09.473943 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:04:09.588032 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:09.643710 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.646108 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.646798 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.650624 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.650975 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:09.675493 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 20:04:09.703850 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (370.769s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:09.704700 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:09.749822 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.752854 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.753183 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.754959 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.755537 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:09.842988 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.845173 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:09.848754 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.850318 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:09.860929 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:09.877323 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:04:09.877875 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:09.883566 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:04:09.991113 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (371.058s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:10.145989 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.148895 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:10.149940 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.150811 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.152768 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:10.239909 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:10.243331 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.244400 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.245723 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.248490 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:10.256232 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.322729 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (371.388s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:10.324520 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:10.326503 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 20:04:10.339048 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.340130 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.343304 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.345249 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.348999 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:10.417752 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:10.447042 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (371.514s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:10.447868 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:04:10.624002 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:10.639585 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.640357 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.642927 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.642978 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:10.653512 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.741479 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.746089 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.746604 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:10.747437 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.763083 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.839728 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:10.842939 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.845009 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.846555 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:10.847292 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:10.906635 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:10.908711 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (371.974s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:10.909571 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:10.919121 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (371.986s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:11.145365 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.146133 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.146180 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.147991 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:11.148411 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:11.165294 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:04:11.241811 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:11.243681 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.243782 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.244050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.244568 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.257700 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:11.318794 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:11.338748 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.342892 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.344991 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.345238 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:11.349539 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:11.375486 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:11.385730 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (372.451s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:11.386473 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:11.435724 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:11.536164 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (372.603s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:11.630648 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.631748 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.632212 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:11.634193 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:11.638672 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:11.652926 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:11.715591 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:11.736466 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.736685 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.737238 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.742477 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:11.746601 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.770830 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (372.836s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:11.772334 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:11.778038 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:04:11.782542 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:11.829437 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:11.830767 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.830840 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.833353 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:11.842682 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:11.940246 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (373.007s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:12.021061 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:04:12.073819 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:04:12.098870 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:04:12.119449 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:12.120352 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:12.134255 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.137461 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.140101 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.140290 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.141525 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:12.229220 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.229657 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.234645 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.240638 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:12.241987 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.288934 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (373.354s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:12.290120 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:12.320196 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.321142 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.323253 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.333098 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.333719 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:12.387380 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:12.447584 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240501 20:04:12.470960 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (373.538s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:12.480334 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:04:12.612811 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.615885 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.617833 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.618410 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:12.627489 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.704707 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.707994 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:12.710107 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.716972 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.720297 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.798413 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.799263 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:12.802901 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.810783 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:12.816807 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:12.822770 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:04:12.846082 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:12.856694 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:12.887748 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (373.953s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:12.888464 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:12.976004 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:13.054836 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (374.122s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:13.055788 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 20:04:13.099483 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.100266 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.102288 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.102428 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.117550 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.193295 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.195552 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.196632 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.198367 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.205065 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.304028 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.304118 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.305656 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.305747 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.306109 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.403834 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (374.469s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:13.404708 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:13.417709 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:13.467748 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (374.535s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:13.572315 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:13.594046 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.597679 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.600371 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.610663 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.613061 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.704715 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.706317 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.708169 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.713407 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.720705 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.799741 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.800583 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:13.803189 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:13.803290 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.804387 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:13.832882 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (374.898s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:13.833822 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:13.952946 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (375.020s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:13.963457 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:04:14.013430 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:04:14.042201 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:14.100040 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:14.100252 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.109200 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:14.111698 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.125170 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:14.174316 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:14.203341 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.203996 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:14.204200 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:14.204303 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.204558 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.236335 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (375.302s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:14.237207 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:14.252367 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:04:14.253245 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:14.297698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:14.298211 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.298403 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.303318 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.318130 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:14.392412 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (375.459s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:14.536412 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:14.597862 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:14.598059 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.598176 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.600538 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:14.609158 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:14.652009 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:14.652550 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:04:14.669656 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:04:14.700510 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.702107 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.704809 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.709081 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:14.717087 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:14.741968 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:04:14.766492 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:14.798379 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.798893 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.800837 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:14.802181 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:14.815060 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (375.880s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:14.815637 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:14.816056 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:14.978425 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (376.045s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:14.978821 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:04:15.017247 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:04:15.018013 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:15.037294 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240501 20:04:15.101735 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.106020 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.107797 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.125094 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.125849 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:15.192243 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:15.203449 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.203811 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.206316 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.206558 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.226792 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.305510 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.308562 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.309161 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.317667 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (376.383s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:15.318702 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:15.319281 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.321238 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.389596 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (376.457s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:15.390544 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:04:15.448453 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:15.494650 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 20:04:15.599953 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.600281 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.603289 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.603873 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.612257 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.700874 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.701021 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.704308 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.709764 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.720098 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:15.739243 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:15.794198 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:15.798015 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.799369 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.801429 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:15.817981 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:15.857789 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (376.923s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:15.858783 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:15.867298 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (376.934s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:15.901145 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:16.052512 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:16.096048 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.098779 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:16.099922 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.113047 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.116889 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.194722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:16.194952 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.199239 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.208864 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.213047 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:16.263664 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:16.264209 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:16.291075 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.295055 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.301229 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.302585 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.307701 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:16.337988 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:16.424346 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (377.490s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:16.425339 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:16.444211 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (377.511s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:16.595981 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:16.596030 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.596513 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.605962 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.608577 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:16.627913 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:16.641440 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:16.647327 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:16.691922 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:16.692579 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:16.694509 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:16.695886 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.699065 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.705992 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.785879 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.787503 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:16.792125 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.795948 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:16.798239 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:16.851506 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (377.919s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:16.893402 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (377.959s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:16.894237 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:16.942070 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:17.040974 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:17.075304 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.079432 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.086581 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.088562 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:17.101653 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.180220 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.180608 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.181488 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.185058 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.191988 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:17.211531 4056 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:17.211633 4056 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:17.212044 4056 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:17.213099 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:17.213166 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:17.213217 10918 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:17.213429 4056 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:17.219298 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:17.219933 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:17.273080 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.278937 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.282792 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.283951 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.289201 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:17.304332 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:17.355542 3844 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (378.421s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:17.356387 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:17.397964 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (378.465s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:17.405195 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:17.415072 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:17.519896 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:04:17.527297 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:17.575883 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.576388 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.576794 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.581086 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:17.585088 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:17.665066 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:04:17.670365 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.670408 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:17.677717 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.679476 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.681869 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.765022 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:17.768508 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.769888 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.770998 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:17.788290 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:17.940541 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (379.008s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:17.940919 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (379.006s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:17.941601 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:04:17.941649 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:18.066149 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.068332 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.068461 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.068758 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:18.072081 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:18.145767 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:18.163926 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.164582 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.167726 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.169787 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.182855 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:18.263478 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.263841 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.265406 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.266362 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:18.266762 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240501 20:04:18.268296 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:18.388862 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:18.471282 7288 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (379.538s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:18.535980 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (379.601s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:18.537035 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:18.555423 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.557394 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.561961 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.566040 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.569326 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:18.648417 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:18.652107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.655540 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:18.657764 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.664803 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.682408 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.754823 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:18.757556 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.759248 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:18.760105 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:18.761868 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:18.789299 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:04:19.029934 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:19.049750 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (380.115s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:19.051443 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.051445 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:19.052209 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:19.053155 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:19.054410 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.055182 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:19.063025 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:19.069805 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (380.137s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:19.145579 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.148499 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.148507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.148758 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.150429 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:19.151099 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:04:19.158210 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:19.242722 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.243424 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.244884 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:19.245839 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:19.246057 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.246649 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:19.328704 6733 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:19.328814 6733 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:19.329142 6733 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:19.329986 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:19.330060 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:19.330144 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:19.330260 6733 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:19.461889 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (380.527s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:19.462811 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:19.479887 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:19.512293 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:19.537650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.538242 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.540035 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.545099 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:19.547443 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.634850 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.634850 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.641489 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.643072 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.650094 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:19.674561 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (380.742s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:19.701118 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:04:19.701743 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:19.731719 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:19.733103 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.734427 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:19.736217 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:19.737834 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:19.741108 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:19.745910 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:19.787191 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:04:19.888552 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:04:20.010799 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:20.028618 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.028771 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.029048 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.032102 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.045511 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:20.053748 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 20:04:20.057520 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (381.125s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:20.082520 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (381.148s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:20.083422 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:20.103520 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:04:20.131415 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.131676 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.137274 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.141276 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.148113 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:20.225121 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.229241 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.230666 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.239715 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.251154 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:20.287662 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:20.460763 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (381.528s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:20.461721 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240501 20:04:20.526175 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.530632 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.530745 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:20.532037 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.538208 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.595268 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (381.660s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:20.596069 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:20.620791 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.624845 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.627727 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:20.636395 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.638495 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.725385 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.725476 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:20.727121 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:20.727351 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:20.734223 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:20.780025 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:20.900230 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (381.967s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:20.928808 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:04:20.957201 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:21.017980 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.019604 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:21.020941 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.022454 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (382.088s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:21.023478 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:21.032106 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.038098 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:21.125826 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.125933 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:21.126677 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.128710 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.144853 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:21.228377 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:21.232007 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.234836 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.237092 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:21.241796 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:21.321831 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:21.391759 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (382.459s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:21.398070 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:21.436549 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:04:21.528568 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.529667 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:21.535324 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:21.537976 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:04:21.542366 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:21.543704 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:21.590493 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:21.622968 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (382.688s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:21.624322 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:21.624955 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.625994 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:21.629444 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.640934 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.641626 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:21.681022 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:21.728817 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.730590 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.732498 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:21.733868 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:21.738194 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:21.858729 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:21.937383 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:21.975066 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (383.042s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:22.027328 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.030048 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.032478 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.033191 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.043396 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:22.071188 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:04:22.075357 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:22.084818 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:04:22.103902 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:22.125200 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 20:04:22.130470 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.131336 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.135625 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.145112 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.147868 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.186084 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (383.251s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:22.186990 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:22.227227 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.231294 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.232522 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.235877 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.245296 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:22.317466 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:22.357898 7459 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:22.357981 7459 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:22.358295 7459 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:22.359122 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:22.359198 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:22.359261 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:22.359438 7459 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:22.395953 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:22.451243 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:04:22.521409 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (383.588s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:22.522975 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.523871 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.525560 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.528225 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.542066 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:22.545512 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:22.579300 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:04:22.617539 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.619271 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.621286 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.622511 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.626551 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.690549 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (383.756s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:22.691756 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:22.714677 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:22.715173 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.718458 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:22.718763 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:22.720517 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:22.747401 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:23.013453 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:23.013824 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.016752 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.018641 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.023193 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.110839 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.115932 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:23.117591 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.129395 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.140486 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (384.207s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:23.141551 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 20:04:23.142139 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.214372 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:23.222693 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.227880 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.229240 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.245646 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:23.265271 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:23.297650 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (384.363s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:23.298612 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:23.373646 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:23.471431 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240501 20:04:23.521742 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:23.527096 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.529228 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.529807 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.530742 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.541450 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (384.608s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:23.616722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:23.620326 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.623024 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.623395 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.625087 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.679930 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (384.745s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:23.680863 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:23.713968 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.716428 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:23.717341 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.726769 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:23.727548 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:23.788199 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:23.899801 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:04:23.986475 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:24.009600 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.010375 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.010974 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.012061 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.021055 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.106993 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.107401 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.108283 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.109728 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.113710 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.126520 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (385.193s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:24.168969 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:24.184403 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:24.204097 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.204387 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.205214 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.206027 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.207110 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.229137 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (385.294s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:24.230090 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:24.256670 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:04:24.307837 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:04:24.503612 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.504729 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.504869 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.507424 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.511775 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.536617 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (385.604s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:24.551432 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:04:24.557750 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:24.564507 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:24.600414 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.600416 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.600647 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.604161 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.604423 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:24.642889 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:24.685523 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (385.751s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:24.686513 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:24.696267 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.698782 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.702491 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.703227 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.707825 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:24.731628 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:04:24.829746 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:04:24.889678 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:04:24.944267 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:04:24.945338 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:04:24.989271 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:24.993073 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.994441 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:24.995512 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:24.996585 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:25.035313 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:25.045081 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (386.112s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:25.084455 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.086467 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.086519 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.091109 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.098259 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:25.106575 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240501 20:04:25.107275 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:25.181291 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.181386 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.187543 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.197077 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.199445 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:25.212651 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (386.278s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:25.216403 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:25.447283 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (386.514s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:25.448294 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 20:04:25.481622 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.482133 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:25.485558 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.487476 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.493906 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.577603 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.577826 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.588696 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.588701 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.590133 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:25.671073 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.679610 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.686868 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.687687 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.695919 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (386.761s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:25.698097 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:25.702339 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:25.727751 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:25.876839 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (386.944s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:25.978778 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:25.984540 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:25.985111 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:25.994566 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.001206 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:26.057344 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:26.080228 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.082649 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.084079 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.085945 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.086642 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:26.098110 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:04:26.142786 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:26.185163 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.187115 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.187762 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:26.189208 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.189778 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.244817 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (387.310s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:26.245627 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:26.305509 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:04:26.341902 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:26.382993 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (387.450s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:26.455338 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:26.480373 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.486230 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:26.486552 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.489081 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.498646 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.581530 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.582266 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.582410 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.589906 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.593159 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:26.631234 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:26.667224 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:26.683964 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.689683 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.690672 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.690682 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:26.691367 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:26.771766 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:26.848474 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (387.914s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:26.849536 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:26.863066 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (387.930s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:26.985159 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.985921 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:26.987263 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:26.988879 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:26.989650 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:27.043730 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:27.065150 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:27.082650 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.083211 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:27.086026 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.093925 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:27.096240 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:27.098659 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.184303 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.185612 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.185608 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.197225 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:27.197710 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:27.223212 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:04:27.245595 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:27.267437 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (388.333s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:27.268193 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:27.279558 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:27.357257 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (388.424s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:27.484158 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:04:27.484933 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:27.485582 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.485945 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:27.486526 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.487133 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.488813 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:27.507092 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:04:27.521458 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:27.577010 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.581341 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.582374 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.583873 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.584326 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:27.586133 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:27.675060 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.682143 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:27.682324 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.692253 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:27.696386 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:04:27.708614 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.760848 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (388.828s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:27.761731 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 20:04:27.858628 7459 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (388.924s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:27.859508 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:27.912791 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:27.988219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:27.991344 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:27.991487 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:27.994221 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.001543 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.080540 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.082574 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.084774 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:28.097448 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.098431 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.180104 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.185617 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.185822 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.189085 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.197459 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:28.334313 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (389.401s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:28.398415 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:28.409073 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (389.474s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:28.410005 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:28.478700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.479764 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.484321 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.484756 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.485910 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:28.509198 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:04:28.534751 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 20:04:28.577046 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.577400 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.578497 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.579886 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.593384 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:28.677788 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.685655 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.685803 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.687207 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.688508 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:28.819242 9842 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (389.886s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:28.850100 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (389.915s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:28.850776 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:04:28.851012 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:28.974062 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:28.976393 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.978544 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.983850 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:28.996968 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:29.005579 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:29.011292 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:04:29.061301 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:29.078275 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.079046 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:29.080626 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.086051 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:29.086925 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:29.099766 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:29.186209 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:29.186949 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.187736 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.188958 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.189232 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:29.293241 9885 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:29.293341 9885 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:29.293689 9885 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:29.294579 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:29.294649 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:29.294734 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:29.294871 9885 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:29.327324 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:29.379490 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (390.446s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:29.431376 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:29.453948 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (390.519s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:29.454838 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:29.485358 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:29.486198 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.487787 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:29.488031 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.489620 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:29.513928 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:29.579855 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:29.582139 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.583468 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:29.584777 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.590576 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.672303 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:29.677805 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.681444 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:29.683681 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.684161 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:29.709606 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:29.781462 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (390.849s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:29.792790 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:04:29.888067 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (390.953s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:29.888985 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:29.945259 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:29.975171 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:29.976707 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:29.977237 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 20:04:29.977471 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.985023 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:29.986532 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:29.994685 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:04:29.995347 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:30.063874 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:30.073596 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.077701 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:30.077790 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.086071 4056 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:30.086160 4056 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:30.086534 4056 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:30.088006 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:30.089133 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:30.089192 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:30.089397 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.089455 4056 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:30.097045 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:30.163817 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:04:30.164873 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:04:30.174014 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:30.177206 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.178939 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.179872 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.180423 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.180825 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:30.284375 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (391.350s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:30.285296 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:30.398361 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (391.465s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:30.399420 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:04:30.463572 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:30.470715 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.471199 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:30.471258 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.472945 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.478767 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.559655 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.564817 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.565594 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.572104 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:30.578205 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.659395 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.661523 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:30.670859 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.672140 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.674553 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.882707 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (391.948s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:30.883551 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:30.931119 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (391.998s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:30.965276 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.971058 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.973448 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:30.981993 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:30.982715 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:31.054631 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 20:04:31.068637 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.069746 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.073791 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.077953 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.079178 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:31.090322 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:31.145944 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:31.163455 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.168007 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.169862 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.170842 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.181427 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.341557 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (392.407s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:31.342362 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:31.447379 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (392.514s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:31.459492 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:31.464504 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.466446 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.466905 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.469048 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.473105 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:31.514741 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:31.515350 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:04:31.528725 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:31.558346 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.558938 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.561858 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.562330 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.568331 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:31.589399 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:31.632272 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:04:31.647966 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.648808 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.652032 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.654191 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.679610 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.740113 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (392.805s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:31.740964 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:31.754664 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:31.958492 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.958516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:31.961540 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.962234 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:31.964771 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:31.994482 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (393.061s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:32.061913 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.062701 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:32.062892 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.067720 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:32.068310 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:32.099054 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:04:32.099944 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:04:32.152134 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:32.165529 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:32.166776 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.168982 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.170751 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:32.187072 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:32.206931 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:32.224505 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (393.290s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:32.225430 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:32.341634 9885 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:32.341732 9885 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:32.342128 9885 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:32.342963 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:32.343051 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:32.343128 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:32.343250 9885 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:32.397763 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 20:04:32.426064 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (393.493s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:32.465262 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:32.467794 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.468144 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.469170 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.470139 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:32.554982 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.555904 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:32.556797 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.557863 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:32.559281 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 20:04:32.559967 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:32.560777 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:32.653395 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:32.653735 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:32.654906 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:32.654927 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.657734 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.658732 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:32.709714 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:04:32.748052 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:04:32.771486 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:32.786386 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (393.852s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:32.787211 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:32.879828 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (393.947s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:32.880816 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 20:04:32.953801 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:32.956593 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.957984 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.959236 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:32.960438 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.048966 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.050472 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.057364 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:33.059429 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.059733 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.146641 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.148502 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:33.148999 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.150354 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.168152 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.209179 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (394.274s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:33.210081 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:33.275169 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:33.284265 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (394.351s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:33.301345 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240501 20:04:33.421460 4056 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:33.421559 4056 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:33.421923 4056 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:33.423039 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:33.423117 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:33.423193 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:33.423331 4056 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:33.447707 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.449698 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:33.451179 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.453208 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.453955 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.536582 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.540831 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:33.544255 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.544868 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.553822 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.635211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:33.636286 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.644749 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.646657 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.647318 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:33.680176 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:33.703464 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (394.771s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:33.738693 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:33.749311 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (394.815s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:33.749995 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:33.813215 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:04:33.835494 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:33.942229 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.942960 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.943560 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:33.945470 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:33.946142 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.035429 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.036880 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.037789 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:34.038606 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.045068 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.092792 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (395.160s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:34.095003 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:04:34.122606 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:34.128214 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.130057 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.134680 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.138020 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.148036 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:34.160699 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (395.226s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:34.162129 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:34.304313 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:34.367079 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:34.423516 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.425904 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.427685 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.430920 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.437359 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:34.492172 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 20:04:34.519439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.519719 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.530714 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:34.534104 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.538616 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:34.580348 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:34.617586 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.619614 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.620074 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.620873 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:34.629412 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:34.633859 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.659143 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (395.726s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:34.670581 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:34.703866 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (395.769s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:34.704603 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:34.832765 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:34.918026 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.918529 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.918727 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:34.918781 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:34.921175 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:35.009727 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.010241 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:35.010444 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.017083 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.028512 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:35.106957 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:35.112241 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:35.113293 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.114609 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:35.116160 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:35.122040 11311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:35.122599 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:35.125793 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.156373 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (396.223s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:35.156946 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (396.222s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:35.157318 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:04:35.157613 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:35.237282 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:04:35.237831 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:04:35.407838 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.408265 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:35.408563 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.421175 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.421341 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:35.441663 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:35.502013 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:35.502727 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:35.503022 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.504612 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.506117 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:35.513200 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.592653 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:35.593221 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.594053 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (396.661s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:35.594180 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.600090 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.605755 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:35.670435 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240501 20:04:35.772500 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (396.838s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:35.773458 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:35.895961 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.896739 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:35.897336 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.897563 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:35.899505 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:35.955238 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:35.992477 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:35.995215 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:36.000797 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:36.001749 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.003787 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.006101 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:36.024081 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:36.052590 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (397.120s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:36.098918 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.099326 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.102058 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.102722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:36.112097 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:36.120736 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:04:36.357443 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:36.394024 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (397.459s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:36.397572 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:36.409456 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:36.410270 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.413940 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:36.414275 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.417326 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:36.479096 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:04:36.496507 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:36.518296 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:36.522785 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:36.523550 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240501 20:04:36.526881 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (397.594s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:36.529392 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.529904 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.530261 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:36.614828 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:36.615119 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:36.617966 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.618642 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.620312 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.851971 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (397.917s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:36.853098 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:36.897258 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:36.911535 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:36.915706 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:36.916061 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:36.921707 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:36.923034 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:36.924536 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:36.953593 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:36.965605 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 20:04:37.006768 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.009220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.010066 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.010308 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.019181 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:37.037957 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:37.096138 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (398.163s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:37.102939 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.102962 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.103639 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.114184 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:37.123633 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:37.355578 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:37.391952 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (398.457s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:37.393276 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:37.397382 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.399652 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.402148 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.414532 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.415094 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:37.490896 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.494962 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.495553 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:37.496855 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:37.502338 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.513887 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:37.536844 4056 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (398.604s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:37.537695 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:04:37.573588 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:04:37.590492 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.592666 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.594657 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.595705 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.602708 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:37.612546 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:04:37.773973 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:37.774574 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:04:37.808072 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:37.889333 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.890087 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.894385 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.901356 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:37.909508 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.984167 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:37.988843 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.989660 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:37.993191 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (399.058s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:37.994405 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:37.996196 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:37.996548 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:38.018692 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (399.086s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:38.045428 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:38.085511 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.087245 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.087908 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.101162 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.105039 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:38.347123 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 20:04:38.389979 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.390363 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:38.390735 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.391453 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.391628 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.469295 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (399.536s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:38.477308 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.479673 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.479918 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:38.482860 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:38.489939 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.490152 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:38.541121 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (399.606s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:38.541962 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:38.575004 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.575430 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.576807 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.577296 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:38.577607 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:38.618173 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:38.621743 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:04:38.666421 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:38.866564 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.867229 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:38.867666 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.869247 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.889060 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:38.948489 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:38.965000 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:38.967283 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.972501 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.974056 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:38.978302 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:39.013486 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:39.058647 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (400.123s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:39.061105 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:39.062747 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.066699 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.066818 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.068152 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.077934 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:39.085471 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (400.152s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:39.126724 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:04:39.175123 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:39.364290 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.365272 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.365535 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.365643 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.367635 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:39.399154 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:04:39.435547 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (400.501s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:39.436690 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:39.451419 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.453655 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.455209 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.460072 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.460178 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:39.541093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.546131 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.546649 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:39.553269 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.554246 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (400.621s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:39.561880 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:39.571780 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:39.596185 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:04:39.635277 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:04:39.825784 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:39.837750 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.843716 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.845042 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.845604 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.846347 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:39.939342 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:39.941449 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.942541 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:39.943713 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:39.944041 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:39.951712 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:40.032274 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (401.097s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:40.034466 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:40.038776 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.039743 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.042126 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:40.042866 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.044088 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:40.069579 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:40.094370 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (401.161s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:40.095321 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:04:40.116391 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:04:40.251289 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 20:04:40.251806 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:04:40.260479 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:04:40.332372 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.332962 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.333230 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.335283 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.344728 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:40.422196 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.422235 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.427711 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.433719 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.441823 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:40.462774 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:40.516790 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.521963 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.524313 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.524540 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.524662 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:40.549336 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:40.622666 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (401.690s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:40.628732 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (401.694s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:40.629519 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:40.785133 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240501 20:04:40.813513 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.813892 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.816254 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:40.817278 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.821316 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.908178 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.911367 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:40.913414 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.921532 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:40.927528 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:41.009989 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.011986 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.018317 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.019155 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:41.024041 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:41.030061 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:41.030678 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:04:41.117473 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:41.127912 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (402.195s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:41.178110 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:41.183741 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (402.249s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:41.184589 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:41.314507 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.316529 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.317195 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.317368 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.326117 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:41.407963 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.409798 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.409895 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.411084 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:41.411978 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:41.480083 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:41.496332 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.499529 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.503744 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.510152 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.514016 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:41.524082 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:41.545863 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (402.613s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:41.554857 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:04:41.771816 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:41.785828 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (402.851s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:41.787070 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:41.796589 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.799225 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.801321 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.808677 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.817500 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:41.831210 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:41.891240 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:41.897063 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.898474 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.903025 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.917568 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:41.946138 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:41.993152 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:41.993688 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.003562 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:42.004206 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.011170 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:42.041603 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:42.074957 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (403.142s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:42.167899 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 20:04:42.286350 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (403.351s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:42.288168 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:42.295729 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.296466 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.296545 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:42.308079 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:04:42.309151 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.311815 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:42.335588 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:42.397799 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.400507 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.401026 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.408706 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:42.409555 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.487018 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.490031 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.490540 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:42.494850 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.502084 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:42.532680 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:42.559517 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (403.627s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:42.560416 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 20:04:42.665791 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (403.731s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:42.666599 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:42.686780 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:04:42.758520 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:42.774217 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:42.780325 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.783334 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.784279 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.789927 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:42.797590 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:42.838883 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:42.839517 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:42.876302 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.882064 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.884853 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.887566 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.890988 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:42.941949 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:42.950471 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (404.018s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:42.970010 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:42.974282 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.975507 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:42.977026 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:42.982182 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.098341 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (404.164s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:43.099087 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:43.158196 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240501 20:04:43.181039 14316 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:43.181135 14316 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:43.181466 14316 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:43.182296 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
W20240501 20:04:43.182456 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:43.182502 11050 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:43.182598 14316 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:04:43.262102 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.264870 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.266399 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.268213 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.274693 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:43.314637 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:43.328022 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (404.395s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:43.330835 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:43.359663 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:43.362432 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.362829 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.363315 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.363759 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.364168 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.454998 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.456456 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.460706 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.470614 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.470990 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.699294 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (404.764s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:43.700799 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:43.708493 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:43.749047 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.750063 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.750329 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.754060 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.757089 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:43.825877 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:43.841079 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.843911 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.845474 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.850015 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (404.917s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:43.851025 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:43.853271 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:43.854888 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.940820 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.942201 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:43.942436 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:43.942920 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:43.957449 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.135797 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (405.201s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:44.136654 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:44.240216 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.243348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.243880 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.245617 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:44.257062 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.271528 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (405.339s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:44.310609 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:44.330220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.332540 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.334312 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:44.341311 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.350116 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:44.388721 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:44.424124 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.426837 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.429227 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.429315 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.434049 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:44.435799 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:44.601974 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:04:44.640298 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:04:44.729454 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.732309 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.733740 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:44.741552 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.742511 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.748116 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (405.813s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:44.754420 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:44.794579 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (405.862s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:44.795475 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 20:04:44.825793 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.827170 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.828647 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.837050 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:44.837050 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:44.883268 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 20:04:44.924705 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:44.928004 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.934914 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:44.941574 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:44.941624 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:44.962392 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:04:45.012156 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:45.135596 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:04:45.175642 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:45.187256 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (406.252s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:45.188520 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:45.217145 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.224731 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.227211 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.229990 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:45.237025 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.275473 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (406.343s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:45.275988 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240501 20:04:45.318594 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.320173 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.321280 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:45.325042 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.325165 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:45.333619 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:04:45.410992 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.413210 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.414227 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:45.421633 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:45.422922 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.426108 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:45.610620 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:04:45.663513 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:45.707875 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.708621 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.711257 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.720366 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.729328 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:45.732403 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (406.799s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:45.760514 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:45.772543 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (406.838s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:45.773545 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:45.805027 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.805493 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.808373 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.810614 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.818501 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:45.820045 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:04:45.838402 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:45.897528 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:45.902956 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:45.903211 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.903728 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:45.904673 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:45.914520 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.125411 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (407.192s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:46.198402 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.199858 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.201728 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.204855 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.213227 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:46.270493 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (407.336s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:46.271381 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:46.302186 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.302925 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.303547 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.303751 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:46.303967 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:46.353140 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:46.392951 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.394814 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.396051 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.397986 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.400750 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:46.409104 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:46.513864 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:46.609609 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (407.677s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:46.683369 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (407.749s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:46.684324 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:46.697122 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.699836 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:46.701659 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.703500 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.710064 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.786645 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.787779 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.789258 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.794013 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:46.801695 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:04:46.802600 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:46.815464 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:46.823556 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:04:46.840601 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:46.889118 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:46.891026 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.892134 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:46.892231 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:46.892987 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:46.933745 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:47.050961 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (408.118s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:47.051926 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 20:04:47.181079 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.181432 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:47.182621 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.187181 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.199509 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.263759 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (408.329s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:47.264637 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:47.284173 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.285059 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:47.287190 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.287940 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.295161 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:47.311074 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:04:47.334883 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:47.374228 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.375079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:47.376426 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.377781 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.381165 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:47.462652 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:04:47.482070 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (408.549s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:47.500562 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 20:04:47.671119 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.677522 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.679723 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.680788 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.681151 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:47.722306 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:47.772971 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:47.774246 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.776118 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.777587 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.793851 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.864059 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (408.929s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:47.865624 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:47.869836 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:47.870756 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:47.871261 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:47.878212 10099 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (408.945s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:47.878971 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:47.884529 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:47.885466 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:47.893570 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:47.896715 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:04:48.030941 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:48.162755 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.164855 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.165297 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.166004 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.169342 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:48.172748 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:04:48.174986 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:48.222090 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:48.263224 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.263485 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:48.265314 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.266088 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.277200 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:48.355186 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240501 20:04:48.355818 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.357486 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.359036 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.359369 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.367337 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:48.386843 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (409.452s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:48.387594 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:48.481798 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (409.549s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:48.575506 13338 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:04:48.575599 13338 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:04:48.575914 13338 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:48.576680 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:48.576753 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:48.576824 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:04:48.576936 13338 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:48.592991 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:04:48.648488 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.649612 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.651690 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.651984 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:48.652344 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:48.717789 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:48.748382 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.752269 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.754210 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.761624 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:48.772675 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.831449 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (409.897s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:48.832378 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:48.845767 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.848608 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:48.850225 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.857306 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:48.868908 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:48.999455 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:49.083837 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (410.151s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:49.117159 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:49.150290 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.152031 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.154428 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.154991 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:49.162994 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:49.204720 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:49.229879 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:49.248929 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.249991 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.250483 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.257494 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.258709 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:49.347362 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.349467 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.351331 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.354298 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:49.356375 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.381206 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (410.446s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:49.382570 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:49.475627 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:49.485203 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:04:49.610818 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:04:49.639524 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:49.639961 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.643158 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.645702 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.647809 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.649250 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (410.716s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:49.650208 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 20:04:49.655961 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:49.682305 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:04:49.743667 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.745846 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.746815 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.757103 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.760334 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:49.835109 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:49.839646 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.845037 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.845031 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:49.848960 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:49.988886 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (411.054s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:49.989691 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:50.130481 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.133690 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.134158 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.136014 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:50.143965 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:50.146633 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:50.224078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:50.227921 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.229596 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.230145 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.230345 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:50.247903 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (411.315s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:50.265439 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 20:04:50.315838 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:50.316450 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.321494 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:50.324654 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.335115 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:50.397912 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:04:50.398452 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:50.491963 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:04:50.497205 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:04:50.528376 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (411.594s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:50.529127 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:50.546680 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:50.568506 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:50.607281 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:50.613284 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.615094 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.624457 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:50.625931 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.662513 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (411.730s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:50.706442 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:50.708972 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:04:50.709980 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.712162 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.713250 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.717087 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:50.775682 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:50.792759 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 20:04:50.800902 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:50.805405 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.806054 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.806674 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:50.813581 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:50.974977 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (412.040s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:50.975734 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:51.102707 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.104336 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.108412 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.117785 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.119400 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (412.186s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:51.121217 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:51.121268 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.209203 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.212159 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.212604 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.224997 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.225574 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:51.284375 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:51.306846 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.308974 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.310079 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.314363 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.314824 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.363226 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (412.428s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:51.364049 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:51.503403 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:51.602041 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.602219 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.603940 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (412.671s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:51.606045 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.606338 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.607718 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:51.633267 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:51.660090 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:51.695398 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:04:51.701967 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.702963 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.703616 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.708050 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.722007 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.797050 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:51.799201 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.800904 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.800928 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:51.805949 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:51.834455 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (412.900s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:51.835529 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:51.866387 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:51.901957 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:52.055076 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:52.057876 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:04:52.081164 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:52.091607 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:52.094956 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.101850 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.103068 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.103619 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:52.116672 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.193723 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (413.261s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:52.195219 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240501 20:04:52.202898 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.207720 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.208472 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.209234 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:52.217171 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.303808 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.304602 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.307410 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.307456 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:52.307456 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.416438 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (413.482s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:52.417331 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:52.586781 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:52.594604 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.594600 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.597633 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.600394 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.605640 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:52.700371 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.701604 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.702004 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (413.769s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:52.703279 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.706267 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:52.715034 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.793269 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:52.793505 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.795327 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:52.796244 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:52.804268 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:52.804352 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:52.935039 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:52.945153 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (414.010s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:52.946151 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:52.947016 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:04:52.954658 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:52.955205 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:53.064296 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:53.086205 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.088279 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.093375 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:53.096412 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:53.111527 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.112133 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.161705 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (414.229s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:53.193104 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.197675 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.197927 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.200285 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.207751 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:53.216318 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:53.235925 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 20:04:53.296483 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.300107 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.300630 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.301483 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.302512 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.404402 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (414.470s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:53.405157 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:53.554628 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (414.622s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:53.592128 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.592417 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.596761 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.601995 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.612128 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:53.674047 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:53.699252 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.702225 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.703022 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.710417 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.712179 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.793220 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:53.793479 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.793836 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.796289 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:53.807909 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:53.826241 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (414.891s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:53.827208 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:53.975805 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (415.043s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:54.005106 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:54.017359 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:54.092700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.093839 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.094461 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.095902 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.096712 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:54.181677 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:54.196223 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.196413 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.205968 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.209082 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:54.213608 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:54.250869 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:54.270107 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:54.297001 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.297871 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.301568 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.306396 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:54.314488 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:54.339179 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:54.408861 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (415.476s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:54.409245 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (415.474s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:54.410185 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:54.410192 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:04:54.556483 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:04:54.567117 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:54.586251 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:54.590173 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.591920 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.594992 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:54.594982 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.597062 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:54.617787 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:04:54.700095 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.701265 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.703845 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:54.707095 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.713769 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.796999 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.799093 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:54.799679 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.810321 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:54.822285 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:54.903255 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (415.970s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:55.028980 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (416.094s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:55.029737 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:55.103649 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.107931 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.113859 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:55.118449 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.119542 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:55.136059 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:55.204775 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.204845 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.205592 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.207767 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.218886 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:55.263674 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240501 20:04:55.302909 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.303969 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.304546 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.304992 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:55.315511 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.391153 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (416.458s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:55.406800 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:55.472600 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:04:55.549037 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:04:55.559515 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:04:55.579865 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:04:55.580569 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:55.594233 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.596148 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.597672 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.601027 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:55.613618 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.625126 13338 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (416.690s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:55.625979 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:55.699208 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.699496 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.704295 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.709738 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:55.710268 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:55.711228 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:55.725486 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:04:55.746944 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:55.789525 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.789696 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:55.790503 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.799939 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:55.800227 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:55.814966 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (416.882s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:56.088438 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:56.089783 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.093048 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.101696 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:56.110282 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.163525 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (417.229s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:56.164489 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:56.169945 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:56.192871 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.194077 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:56.196942 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.198112 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:56.205246 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.293079 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.296859 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.302616 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.302796 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:56.303009 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:56.421787 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:56.435614 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (417.503s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:56.581358 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (417.646s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:56.582650 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:56.589716 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:56.591440 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.592787 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.595903 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.605070 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:56.621712 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:56.696959 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:56.697180 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.699976 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:56.701277 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:56.706893 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:56.713068 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.789605 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.792263 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:56.793160 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.795121 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:56.795254 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:56.879961 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:56.912545 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (417.980s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:56.913529 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:04:56.915336 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:04:56.949498 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:04:56.965097 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (418.030s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:56.965821 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:56.968761 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:57.087181 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.092569 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.098742 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.105185 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:57.113042 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:57.125101 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:04:57.129402 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:04:57.147845 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:57.201061 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.201146 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.202190 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:57.205956 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:57.213172 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:57.262179 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:57.300603 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:57.302919 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.304003 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.305028 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.315433 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:57.370788 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (418.438s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:57.535189 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (418.600s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:57.536026 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:57.539088 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 20:04:57.598878 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:57.600569 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.603322 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.603730 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.605080 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:57.685086 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:57.704797 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:57.707240 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.707479 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.707854 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.717279 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:57.759011 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (418.826s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:57.795651 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:57.796052 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:57.796870 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.799130 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.805686 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:57.806106 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:57.928298 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (418.993s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:57.929222 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:57.940548 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:04:57.949527 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:04:57.992959 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 20:04:57.993590 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:58.089112 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.089993 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.095278 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.103056 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.108565 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:58.125494 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:04:58.126232 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:58.193063 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.193837 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.196808 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.205055 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.207060 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:04:58.246201 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:58.293303 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:58.295917 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.296751 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.297626 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.309670 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.330277 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (419.397s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:04:58.421975 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (419.487s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:58.422855 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:04:58.588984 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.590487 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.590731 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.591140 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:58.597932 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.603425 10830 maintenance_manager.cc:419] P 06cf8280f20843729e99a32633f62d20: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033378
I20240501 20:04:58.611670 10829 maintenance_manager.cc:643] P 06cf8280f20843729e99a32633f62d20: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.002s sys 0.000s Metrics: {"bytes_written":2221,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":70,"threads_started":1,"update_count":36,"wal-append.queue_time_us":5193}
I20240501 20:04:58.612026 10830 maintenance_manager.cc:419] P 06cf8280f20843729e99a32633f62d20: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.151842
I20240501 20:04:58.619406 10829 maintenance_manager.cc:643] P 06cf8280f20843729e99a32633f62d20: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14983,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":137,"lbm_read_time_us":236,"lbm_reads_lt_1ms":17,"lbm_write_time_us":237,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":94,"threads_started":2,"update_count":36}
I20240501 20:04:58.692159 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:58.692243 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.697937 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.708597 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.712109 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.738701 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (419.806s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:58.750340 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:58.794759 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.798758 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:58.800454 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.805507 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:58.805961 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:58.864434 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (419.930s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:58.865371 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:59.013684 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:04:59.073081 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:59.096681 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.097539 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:59.098311 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.099186 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.104336 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.195588 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.196990 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.199239 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.199328 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (420.266s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:59.200631 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 20:04:59.201823 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:59.216419 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:04:59.259735 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:04:59.283851 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:04:59.299762 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.300175 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.300587 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.309651 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:59.311327 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.350404 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (420.416s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:59.351320 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:59.368459 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:04:59.594221 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:59.595798 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.596279 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.599445 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.600046 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.611624 18394 raft_consensus.cc:482] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 417972166f4d40ca9b4b6471a90be6af)
I20240501 20:04:59.611726 18394 raft_consensus.cc:504] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92ecfb9f8f6745efa5aa2e168bafcc4e" member_type: VOTER last_known_addr { host: "127.30.1.66" port: 38739 } } peers { permanent_uuid: "ea26a64181784350bcd3540d0633b219" member_type: VOTER last_known_addr { host: "127.30.1.71" port: 32843 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
W20240501 20:04:59.612700 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:59.614897 18394 leader_election.cc:288] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:04:59.616362 11050 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:04:59.616495 11049 leader_election.cc:334] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:04:59.616552 11049 leader_election.cc:302] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92ecfb9f8f6745efa5aa2e168bafcc4e; no voters: 417972166f4d40ca9b4b6471a90be6af, ea26a64181784350bcd3540d0633b219
I20240501 20:04:59.616997 18394 raft_consensus.cc:2726] T c49b9c5ebaab42e18fe7bbe8127d569e P 92ecfb9f8f6745efa5aa2e168bafcc4e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:04:59.637321 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:04:59.700274 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.700748 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.704584 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.705628 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.713060 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:04:59.727922 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:04:59.742420 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:59.753585 17727 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (420.821s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:59.754694 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:04:59.794487 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.797031 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.798429 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:04:59.800253 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:04:59.801777 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:04:59.956967 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (421.022s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:04:59.957801 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:00.000774 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240501 20:05:00.093371 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:00.109012 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.109333 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.113013 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.117074 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:00.159382 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:00.205343 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:00.205444 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:00.205636 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:00.205710 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.207263 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:00.213414 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:00.219429 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:00.246152 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:00.253158 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (421.320s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:00.286947 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:00.302995 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.303460 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.305042 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:00.305393 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.315850 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:00.378878 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:00.497999 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (421.563s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:00.498836 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:00.594775 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.596117 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.596772 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:00.599138 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:05:00.600415 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:00.602187 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:00.603478 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.698359 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:00.703744 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.704551 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.705312 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:00.713976 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:00.715613 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:05:00.729244 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 20:05:00.792835 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:00.796298 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.796330 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.798468 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:00.801870 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:00.845240 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (421.912s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:00.995291 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (422.060s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:00.996155 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:01.087808 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.090409 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.090430 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.099686 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.100462 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:01.167950 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:01.196074 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.197377 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.201695 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.209728 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.220175 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.306308 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.307147 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.309165 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.309278 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.318029 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.401947 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (422.469s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:01.576756 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:01.597505 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.600046 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.602270 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.604146 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.609068 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.612947 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (422.678s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:01.613734 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:01.698896 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.700557 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.701046 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.706694 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.709292 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:01.722025 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:05:01.735224 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:01.744365 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:01.790736 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.792241 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:01.792608 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.793056 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:01.802318 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:01.891337 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (422.958s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:01.892261 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:05:02.015033 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:02.026154 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:05:02.064211 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:02.081274 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:05:02.081929 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.082019 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.084738 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:02.087973 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.093304 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (423.159s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:02.093680 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:02.094319 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:05:02.095595 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:02.167274 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:02.186904 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.187438 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.188913 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.188980 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.196246 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:02.281658 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.282991 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.285557 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.291676 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (423.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:02.292974 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:02.292990 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:02.336863 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:05:02.384258 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 20:05:02.547638 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (423.613s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:02.548487 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:02.576256 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.576771 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.577565 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.589645 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.590361 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:02.635038 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:02.673697 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.676602 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:02.676937 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:02.678390 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.683977 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.696373 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:02.753247 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:02.773373 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (423.840s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:02.774075 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.777655 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:02.779753 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:02.783583 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:02.791606 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:02.801863 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:02.960433 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:02.996891 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:05:02.997560 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:03.000082 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (424.065s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:03.000974 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:03.068797 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.073758 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.075752 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.077622 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.085084 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:03.161151 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240501 20:05:03.173566 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.176252 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:03.181054 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.187089 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.187736 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.273268 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.273337 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.275537 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.281776 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:03.293891 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:03.294772 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:03.381846 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (424.449s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:03.519078 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (424.584s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:03.520052 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:03.577843 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.578478 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.579519 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.580693 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.589624 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:03.669857 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.672752 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:03.692236 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.693344 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:03.700136 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:03.702390 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.778761 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.782683 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.783849 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:03.784260 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:03.789045 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:03.872524 18662 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (424.940s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:04.081722 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.083902 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.084597 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:04.093739 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.093952 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.130151 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (425.195s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:04.131965 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:04.147465 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:04.179991 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.184406 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:04.189119 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.189388 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.197819 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:04.209656 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:04.215279 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:04.267673 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (425.335s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:04.268831 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240501 20:05:04.278748 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.278770 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.279075 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:04.281409 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.289090 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:04.305785 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:05:04.357100 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:04.357472 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:04.442546 10827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:05:04.443161 10827 catalog_manager.cc:4573] TS ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b4f0699b5bdf44499c87ff1541cd2ad3: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111)
I20240501 20:05:04.509608 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (425.575s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:04.510596 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:04.571555 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.571827 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.571884 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.572067 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.579326 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:04.579737 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:04.614167 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:04.668639 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.675779 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.680389 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:04.684695 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:04.685684 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:04.686076 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:05:04.705351 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.777441 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (425.844s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:04.779618 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:04.782426 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.783318 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:04.783526 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:04.794941 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:04.972851 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:05:04.979847 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:05:05.013247 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (426.078s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:05.014300 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:05.074532 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.079658 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.080279 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:05.087425 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:05.105334 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:05.108469 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.194922 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.195326 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.196502 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:05.198252 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:05:05.198560 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.198865 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:05.242206 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:05:05.254918 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (426.322s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:05.297657 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.298934 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.299124 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.299145 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.302506 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:05.326261 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:05:05.390962 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:05:05.401086 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (426.466s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:05.401865 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:05.511955 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:05.585783 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25ea8d7ab1834abfa34ff2357aa554b0: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:05.593962 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.594233 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.595417 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.597126 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.598665 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:05.636715 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (426.704s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:05.637959 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:05:05.693820 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:05.695895 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.697527 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.703308 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.703739 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.712306 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:05.791282 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:05.794224 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.796209 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:05.799719 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:05.800748 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:05.823251 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:05:05.914515 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (426.980s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:05.915320 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:06.058755 19761 raft_consensus.cc:482] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3c6b84864cb4358a9d7e0f2a65ba83d)
I20240501 20:05:06.058872 19761 raft_consensus.cc:504] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "99348cf9a56645808d00c459fb76a6f8" member_type: VOTER last_known_addr { host: "127.30.1.65" port: 40027 } } peers { permanent_uuid: "e3c6b84864cb4358a9d7e0f2a65ba83d" member_type: VOTER last_known_addr { host: "127.30.1.72" port: 35171 } } peers { permanent_uuid: "417972166f4d40ca9b4b6471a90be6af" member_type: VOTER last_known_addr { host: "127.30.1.69" port: 38171 } }
I20240501 20:05:06.059336 19761 leader_election.cc:288] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171), 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171)
W20240501 20:05:06.060204 10918 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:05:06.061224 10919 leader_election.cc:334] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:06.061304 10919 leader_election.cc:302] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99348cf9a56645808d00c459fb76a6f8; no voters: 417972166f4d40ca9b4b6471a90be6af, e3c6b84864cb4358a9d7e0f2a65ba83d
I20240501 20:05:06.061504 15753 raft_consensus.cc:2726] T a170766f497f491ba9b45f92f2a0e751 P 99348cf9a56645808d00c459fb76a6f8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:05:06.088495 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (427.155s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:06.092192 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.092425 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:06.092445 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.097723 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.098162 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.193926 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.196897 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:06.197090 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.197458 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.219928 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:06.248971 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:06.299149 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.299871 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.301731 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:06.303766 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.306792 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.321137 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (427.386s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:06.329082 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:06.390019 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:05:06.582326 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (427.649s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:06.583245 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:05:06.595034 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:06.612619 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.615240 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.615412 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.615689 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.616829 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:06.659219 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:06.712494 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.716941 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.718044 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.719645 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.721195 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:06.730407 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:06.734808 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:06.762442 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:06.809348 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:06.811905 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.812058 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:06.814033 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:06.817988 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (427.883s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:06.820757 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:06.833124 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:07.006705 10828 catalog_manager.cc:4573] TS 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbc6ed9f4bf34b1caa462aa9d85c7a16: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:07.066058 11180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:05:07.108798 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.108865 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (428.176s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:07.111150 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:07.114387 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.128430 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.141104 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:07.142364 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:07.196611 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:07.215034 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:07.220005 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:07.221171 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.227970 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.233018 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:07.236042 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:07.294131 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (428.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:07.295663 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:07.317297 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.317806 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:07.319729 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:07.321462 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:07.321805 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:07.330865 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:07.474072 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:05:07.593744 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240501 20:05:07.615857 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.619500 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:07.620949 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.621150 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.637204 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:07.666045 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:07.692822 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (428.758s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:07.694051 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:07.719516 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (428.786s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:07.720657 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.720700 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:07.734140 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:07.737521 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.738019 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:07.738055 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.826957 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.827212 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:07.827956 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:07.831323 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:07.840227 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:07.841158 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:08.028507 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:05:08.078667 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:08.094769 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (429.160s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:08.098150 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:05:08.099117 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:05:08.101626 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:08.123773 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:08.125149 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.130584 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.134461 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.134577 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.145666 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.227820 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.229581 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (429.297s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:08.229974 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.230095 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.242918 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.250552 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.323621 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.326678 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.329679 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.332010 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.343727 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.591408 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (429.657s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:08.592453 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:08.620400 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.622933 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.623059 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.624831 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.627240 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:08.713342 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:08.723435 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.726292 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.726962 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.733067 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.736203 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.738806 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (429.806s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:08.800264 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:08.826162 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:08.827782 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.832473 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.833786 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:08.841428 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:08.978003 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (430.043s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:08.978837 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:09.109720 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:09.126761 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.128259 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:09.136981 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:09.139325 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.148218 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:09.151496 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:09.168516 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:09.184360 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:09.229099 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:09.231209 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.233459 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:09.234850 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.248646 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:09.301348 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:09.330078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:09.330685 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.332846 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.338129 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.354390 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:09.361974 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (430.429s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:09.362818 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:05:09.366792 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:09.444566 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (430.510s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:09.445410 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:09.630148 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:09.631264 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.633469 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.634708 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.636140 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:09.682166 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:09.737226 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:05:09.738703 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:09.741618 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.742075 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.742115 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.744211 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:09.833454 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (430.900s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:09.836231 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.836958 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.839412 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:09.841717 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:09.848078 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:09.848134 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:05:09.921227 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:10.013690 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (431.079s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:10.014590 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:10.142967 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:10.142972 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.143041 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.153404 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.166745 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:10.213145 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (431.280s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:10.243598 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:10.248111 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:10.248682 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.253039 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.257577 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:10.272841 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:05:10.320050 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:10.336769 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:10.339449 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.339929 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.341405 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:10.344746 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:10.375025 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:05:10.387589 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:10.507721 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:10.553294 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (431.619s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:10.554044 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:10.591336 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (431.658s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:10.633100 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.638629 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:10.639396 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.645236 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.645469 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:10.714213 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:10.718429 10918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 75 similar messages]
I20240501 20:05:10.743368 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:10.744809 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.746111 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.753950 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.756260 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:10.805855 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:10.830050 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:05:10.834775 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:10.836383 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.844086 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.844772 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:10.853813 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:10.973152 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (432.038s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:10.973906 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:11.004581 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:11.019523 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (432.087s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:11.127408 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:11.136000 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.142966 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.143054 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.146246 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.158715 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:11.244055 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.244043 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.248634 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.251077 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:11.253597 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.347378 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.349489 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.351688 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.353865 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.354303 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:11.511163 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (432.576s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:11.512091 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:11.516651 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (432.584s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:11.517474 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240501 20:05:11.645268 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.646625 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.648959 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.649693 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:11.651037 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:11.685098 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:05:11.701203 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:05:11.734215 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:11.748149 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.751652 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.752537 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:11.760044 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.762660 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:11.770916 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:05:11.815650 11312 consensus_peers.cc:485] T 5d032fb77ac944629a45b808cc76c2f1 P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:11.850030 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:11.852108 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.852488 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.854794 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:11.857520 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:11.861459 10918 consensus_peers.cc:485] T 9fcd6776aebf47c99f05850de83b36e6 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:11.999224 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (433.064s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:12.000114 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:12.099166 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (433.166s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:12.110217 11181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:12.135629 11049 consensus_peers.cc:485] T 269047cc3a504d7488747166efce71fc P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:12.158669 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:12.160862 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:12.162212 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.165386 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.171474 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:12.189241 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
W20240501 20:05:12.195995 11180 consensus_peers.cc:485] T 19f79cd7d4ec4b4697a2637e6d14a892 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:05:12.256212 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:12.256893 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.261973 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:12.268625 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.269063 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.348553 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.351346 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:12.353387 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:12.353471 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.362161 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:12.468601 11311 consensus_peers.cc:485] T 5eb1f0dcd4cf49cd969eb28f1eb5d9ea P 1679f84270414f67ac67c882220e1472 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:12.491566 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (433.557s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:12.492430 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
W20240501 20:05:12.564870 11181 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:05:12.631795 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:12.647439 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:12.651315 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.652573 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.654137 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:12.655505 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.680361 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (433.747s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:12.753937 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.756983 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.761488 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:12.764606 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:12.765281 11180 consensus_peers.cc:485] T 2d6621e1ff294dc9a474e076873cc72c P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 20:05:12.769068 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:12.808454 10918 consensus_peers.cc:485] T dbc6ed9f4bf34b1caa462aa9d85c7a16 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:05:12.823349 11049 consensus_peers.cc:485] T 219742bd987e4a1990233fa8c972c232 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:12.855058 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:12.855875 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.860437 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.869040 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:12.873055 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:13.079865 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (434.147s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:13.109040 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (434.174s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:13.109946 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:13.160534 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:13.161774 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.164564 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.168182 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.176416 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:13.202175 11049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:13.202800 11049 consensus_peers.cc:485] T 19d8d7d255494a49920849d10b800fa7 P 92ecfb9f8f6745efa5aa2e168bafcc4e -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:05:13.245136 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:13.258613 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:13.259815 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.261565 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.270319 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.273250 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:13.355281 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:13.356055 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.358850 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.358901 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.361249 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:05:13.382422 11312 consensus_peers.cc:485] T 8fc8c5aab87341e3b79a8e121e6db01e P 1679f84270414f67ac67c882220e1472 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Couldn't send request to peer 417972166f4d40ca9b4b6471a90be6af. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:05:13.434949 10918 consensus_peers.cc:485] T 0a1bbe74dbc74fa687b0a64d3aa203f3 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:13.528564 15753 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (434.594s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:13.529511 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer 417972166f4d40ca9b4b6471a90be6af (127.30.1.69:38171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.69:38171: connect: Connection refused (error 111)
I20240501 20:05:13.552125 19761 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (434.619s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240501 20:05:13.650592 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:13.650689 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:13.663499 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.665553 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.665627 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240501 20:05:13.679773 10918 consensus_peers.cc:462] T 8e48d711c5714ee58ebbcbd094ebf3e7 P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111)
I20240501 20:05:13.757323 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.757557 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:05:13.760076 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.765945 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:13.766387 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.853120 11094 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.853178 11354 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.854719 11224 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:05:13.857445 10961 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:05:13.857543 12753 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:05:13.980391 10918 consensus_peers.cc:485] T 2def1cda07c349c69b3abe25804d3d1d P 99348cf9a56645808d00c459fb76a6f8 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:05:14.000357 11180 consensus_peers.cc:485] T 156d2ba7ff2c403db2f63cbae4acfec1 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:05:14.058101 22254 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer ea26a64181784350bcd3540d0633b219 in term 1 (Leader has been unable to successfully communicate with peer ea26a64181784350bcd3540d0633b219 for more than 300 seconds (435.125s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:14.058899 10918 consensus_peers.cc:485] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 -> Peer ea26a64181784350bcd3540d0633b219 (127.30.1.71:32843): Couldn't send request to peer ea26a64181784350bcd3540d0633b219. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.71:32843: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:05:14.148041 11180 consensus_peers.cc:485] T 54d2db0f85ac4a90a1df779f89398463 P 4456bc4917c644f8b69b4f5bfb8e1166 -> Peer e3c6b84864cb4358a9d7e0f2a65ba83d (127.30.1.72:35171): Couldn't send request to peer e3c6b84864cb4358a9d7e0f2a65ba83d. Status: Network error: Client connection negotiation failed: client connection to 127.30.1.72:35171: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:05:14.152895 22254 raft_consensus.cc:952] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 [term 1 LEADER]: Processing failure of peer 417972166f4d40ca9b4b6471a90be6af in term 1 (Leader has been unable to successfully communicate with peer 417972166f4d40ca9b4b6471a90be6af for more than 300 seconds (435.218s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240501 20:05:14.154389 10919 consensus_peers.cc:462] T d5faf889870246b78b82e19c041dc119 P 99348cf9a56645808d00c459fb76a6f8 ->